Managing Go 1.5 Vendor Dependencies with Glide

Aug 20 2015

There is a new way to manage Go dependencies, and Glide 0.5 makes this process efficient and powerful. Glide 0.5 is a feature-packed release supporting:

  • 100% full vendor/ support, including deeply nested projects.
  • Recursive dependency resolution
  • Dependency management aware of...
    • Git, Mercurial, Bazaar, Subversion...
    • Tags, branches, forks, and commits...
    • OS and Architecture dependencies...
    • And Godeps and GPM!
  • And also the ability to exclude vendor/ from common Go operations like running tests.

Go's New Vendoring System (And What That Means for Us)

Go 1.5, released hours ago, includes a new way to manage dependencies. Streamlining an existing technique, the Go tools now treat the directory vendor/ specially. In short, packages included in a project's vendor/ directory are treated similarly to packages installed in $GOPATH/src/. You still have to enable a flag for this experimental feature, but Russ Cox and the Go team have made it clear that vendor/ will be the place to store vendored dependencies.

To enable the experiment, set an environment variable.

export GO15VENDOREXPERIMENT=1

That's all you need to do to use Go's new Vendor Experiment... and Glide's vastly improved vendor managing capabilities. No more GOPATH munging or gliding in and out of projects.

Managing Vendor Directories with Glide

Matt Farina and I originally created Glide to manage Go packages for large projects that had dozens of dependencies. My use case was the Go cloud services that we ran at Revolv (now part of Google).

To meet the demands of these sizable Go projects, Glide used to aggressively manage the $GOPATH. But not anymore. With the 0.5 release, we have re-invented Glide, and made it much better.

Glide continues to use the glide.yaml file to store information about project depenedencies. Specifically, it is used to store information on source code versions and locations. Creating a glide.yaml file is as easy as running glide init. Then Glide works side-by-side with the existing Go tools. No weird wrapping or extra-long commands.

To install a package globally, you can continue to use go get:

$ go get github.com/Masterminds/sprig

But to install a package into your vendor/ directory, use glide get.

$ glide get github.com/Masterminds/sprig

The above will not only install the dependency into vendor, but will automatically store version information in glide.yaml. That means you don't need to store vendor/ in your VCS. Store glide.yaml, and Glide can restore vendor/ into its necessary state simply by running glide up.

Compiling, running, testing, vetting, linting, and so on can all be done with the regular Go tools, no modification needed.

Recursive Dependency Resolution

The new Go Vendor Experiment supports nested dependencies, like this:

github.com/my/project/vendor/bitbucket.com/your/project/vendor/...

Glide can recursively resolve these deeply nested dependencies during a glide get or glide up command. Package conflicts are not the problem they once were. (We're even working on a fix for the elusive side-effect imports problem with SQL drivers!)

Glide Does Godeps!

We know that many projects have relied upon Godeps and GPM. And many will continue to do so.

Rather than ignore these solutions, we added support for them!

glide import godep remains an easy way to transform older Godeps projects into glide.yaml. But we've made it easier to co-exist with projects that use other dependency managers.

Want to import a project that manages its dependencies using Godeps? Piece. Of. Cake.

$ glide up --import

Both glide get and glide up now sport an --import flag that will automatically resolve Godep and GPM files and store those dependencies in the correct vendor/ folder. Our testbed for this feature is the massive Deis codebase, which imports over 70 libraries, of which many of those have their own dependencies. And it all works great!

(Oh, and we've always supported gopkg.in!)

VCSes, Forks, Architectures, and OSes

Glide is designed for projects with multiple developers. So we added some features designed for flexibility:

  • Glide supports the common VCSes: Git, Mercurial, Bazaar, and SVN. The library we use is more powerful than the Go Tools VCS library, so we can offer great additional features, like the ability to determine how far out of date a dependency is.
    • Glide doesn't just understand commit IDs, but also tags and branches. Want a dependency to stay current on a stable branch? Glide makes that a piece of cake.
    • But wait! With the glide pin command, you can take that flexible Glide file and generate a "pinned" version where an exact commit is supplied. That makes it easy to lock down projects when you push to production.
    • And did we mention easy support for private VCS repos?
  • Glide can handle package masking easily, so you can fork a repository without changing the import paths.
  • Glide supports flagging dependencies by platform. Is a library only needed on Windows builds? Glide can ignore that requirement on non-Windows systems, so developer teams don't have to deal with annoying dependency compilation problems. This holds for all Go supported OS and architecture flags.
  • Finally, Glide has been tested on Linux, OSX, and Windows. It even works great inside of Docker containers! That makes it a powerful candidate for cleverly deploying cloud applications.

Avoid Running Tests in vendor/

In our efforts to be 100% vendor aware, we caught on to one inconvenient (yet correct) side-effect of the new Vendor Experiment.

One gotcha with the new Go Vendor Experiment is that the vendor/ directory is still treated as part of the local project. So what happens when you install packages into vendor/ and then run go test ./...?

All of the tests in your project and in vendor/ are executed.

To make it more convenient to work with vendor/ directories, glide added the novendor (or nv) command:

$ go test $(glide nv)

The above will run the tests from the local project, but not from vendor.

Onward To 1.0!

We are really proud of 0.5, and HUGELY enthusiastic about Go's Vendor Experiment. So we're continuing on our way to Glide 1.0. Here are some of the features we're working on:

  • Support for side-effect imports (without the bad kind of side-effects)
  • Full semantic versioning support
  • Easier ways to track how far your dependencies are from upstream
  • Ways to "squash" your dependency trees when nested vendor directories just get too deep or too repetitive

Got more suggestions? Drop an issue in the queue at github.com/Masterminds/glide

And things that went away...

Sometimes it is necessary to remove features. Glide has hit that point. We no longer think $GOPATH management is the way to go. The time has come to say goodbye to Glide's $GOPATH management. So we've removed the following:

  • glide in
  • glide into
  • glide env
  • glide gopath

The glide ZSH plugin is deprecated. ALREADY_GLIDING is gone from the environment. And much of the logic behind resolving project-specific $GOPATH has simply been removed.

We've also made Glide less aggressive about deleting unused dependencies.

The times have changed, and we're excited about a sleeker, more powerful Glide.