Why GPM is the Right Go Package Manager
The debate rages on over which method is best for managing dependencies in Go. But I've found what seems to me to be the right solution to the problem, and it's a tiny little Bash script called GPM.
Go comes with a variety of marvelous and problem solving tools. But one thing it lacks is any built-in mechanism for managing the versions of dependencies. This is compounded slightly by the fact that there is a strong correlation between package names and the source repository for the package. (e.g. the package github.com/technosophos/foo is assumed to live at that URL.)
The Problem with Go Package Management
So how do we make sure that when you build a package that depends on github.com/technosophos/foo and I build one, too, we're building the same thing? What if I'm building against version 1.0, and you're building against the just-this-second released 1.0.1? In short, we want a way to have repeatable builds and known-good dependencies. For brevity's sake, I'll refer to this goal as package management.
There are two dominant theories about how to manage packages in Go.
The first is simple enough: always run head. In other words, update all of your dependencies often, and adjust as necessary. Until recently, this was my preferred route. But a surprise major undocumented change by one of our core dependencies changed my mind. Rewriting a ton of code to meet the new API gave me plenty of time to think about whether we should keep living at the tip of every dependency's source code.
The second theory is called vendoring. The idea is that all of your dependencies go in a particular subdirectory of your app (usually _vendor or vendor), and you manage these dependencies inline. This strategy is used by a broad number of programming languages, including Node.js and PHP's Composer system. But there's an additional catch in Go: since package name is correlated with package location, you have to rewrite the source to replace the package names. And this means not only your source, but the sources of all packages that are vendored. I have many reasons I don't like this approach, but I can summarize it by saying that it feels like a hack that comes with maintenance headaches galore.
GOPATH and GPM
Go makes heavy use of a certain environment variable, $GOPATH. According to the documentation:
The GOPATH environment variable specifies the location of your workspace. It is likely the only environment variable you'll need to set when developing Go code. How To Write Go Code
When I started with Go, I used a single $GOPATH for all of my Go code. But the longer I worked with Go, the more I felt like I was doing it wrong. Maybe I should be using one $GOPATH per project. Once I'd started to think this way, I realized that the right way to manage Go paths is the way that GPM does it.
GPM (Go Package Manager) is a simple tool for managing a project's dependencies by using a special file, Godeps, to declare dependencies and optionally pin the at a specific version number.
For example, here's the beginning of one of my Godeps files:
github.com/Masterminds/cookoo
github.com/aokoli/goutils 1.0.0
github.com/bmizerany/pq 284a1d41dcc27fe8ccc413c339464698ab99f6d1
This declares three dependencies. The first, Cookoo, is always pointing toward head. That's great for me, since I'm a core maintainer on that project, and know its status exactly.
The second, goutils is pinned to a particular Git tag. That library recently had a 1.0.0 release, and I'd like to use just that release.
The third, pq, is our Postgres driver. I have it pinned to the Git commit hash of the version we've currently got installed and tested.
Now if I use gpm install, GPM will first go get all three of those packages, and then set their versions to whatever I have specified. Effectively, GPM is managing my $GOPATH for me.
GOPATH and GVP
The main caveat when using GPM, though, is that each project will need to have its own $GOPATH. (Either that or you have to commit to having all projects agree on dependency versions.)
The developer of GPM also wrote a tool to help manage $GOPATH. It's called GVP (Go Versioning Packager). Basically, GVP creates a .godeps directory inside of your project, and then generates a $GOPATH that includes that directory.
The idea behind GVP is similar in theory to the idea of vendoring, but with two major differences.
- Because the
$GOPATHpoints to the.godepsdirectory, there is no need to modify package paths. - Because (if you use GPM) and exact state can be rebuilt at any time, you do not need to store the
.godepsdirectory in the project. It can be rebuilt on-the-fly.
An Example
To make this clearer, here is an example. I already have both gpm and gvp installed, and my $GOPATH is initially set to my old way of having all projects in one GOPATH.
$ echo $GOPATH
/Users/mbutcher/Code/Go
$ ls
main.go main_test.go
Now let's use gvp to manage and set the $GOPATH:
$ gvp init
$ source gvp in
$ echo $GOPATH
/Users/mbutcher/Code/Go/src/github.com/technosophos/gpm-example/.godeps:/Users/mbutcher/Code/Go/src/github.com/technosophos/gpm-example
The gvp init call creates the necessary infrastructure, and source gvp in sets my $GOPATH environment variable. As you can see, the $GOPATH now points to .godeps and the project's home directory. (I know some people argue vehemently against multiple directories in $GOPATH, but this actually works quite well in the present system.)
Now let's take a quick look at main.go:
package main
import "github.com/aokoli/goutils"
func main() {
println(DoSomething())
}
func DoSomething() string {
r, _ := goutils.RandomAscii(5)
return r
}
Notice that I import one external library (github.com/aokoli/goutils). If I try to run this code, I'll get an error:
$ go run main.go
main.go:3:8: cannot find package "github.com/aokoli/goutils" in any of:
/usr/local/Cellar/go/1.2/libexec/src/pkg/github.com/aokoli/goutils (from $GOROOT)
/Users/mbutcher/Code/Go/src/github.com/technosophos/gpm-example/.godeps/src/github.com/aokoli/goutils (from $GOPATH)
/Users/mbutcher/Code/Go/src/github.com/technosophos/gpm-example/src/github.com/aokoli/goutils
Simply stated, that library is not in my current $GOPATH. I can fix this by creating a Godeps file and then installing with gpm. Here's my Godeps file:
github.com/aokoli/goutils 1.0.0
Now I can run gpm:
$ gpm install
>> Getting package github.com/aokoli/goutils
>> Setting github.com/aokoli/goutils to version 1.0.0
>> All Done
And now the program will run and can be tested:
$ go run main.go
I3!eZ
$ go test
PASS
ok _/Users/mbutcher/Code/Go/src/github.com/technosophos/gpm-example 0.011s
So there we have it. The project is building correctly and repeatably.
When I'm done with the project, I can re-set my $GOPATH to its old default:
$ source gvp out
>> Reverted to system GOPATH.
$ echo $GOPATH
/Users/mbutcher/Code/Go
And now I can go about my other work.
Conclusion
GPM + GVP has worked well for my needs, and I find it much better than vendoring or living at the cutting edge of every dependency's repository. I've implemented it on projects large and small, and it feels clean. I've used it to manage builds and CI. Overall, I've found it to meet my needs better than any other solution.
The system, though, is not without its limitations:
- Building up your initial
Godepsfile can be a little tedious if you begin with a large project. Even thegpm bootstrapplugin only goes so far in bootstrapping. - Like other tools, this does not resolve cases where two dependencies each require different versions of a third dependency. While this hasn't hit me yet, some day it will, and I'm not sure how I'll solve it.
I've hit other minor frustrations, but have been resolving them by writing plugins to GPM like gpm-git and gpm-local.
