Don't Condemn Go To Repeat Past Mistakes
The American philosopher George Santayana famously quipped:
Those who cannot remember the past are condemned to repeat it.
A litany of "history repeating itself" variations have trickled down from Santayana's phrase. But the original captures something that those "history" variants do not: the lesson isn't just about world history. It's about science, too. Including computer science. And the present topic is what Go should do about package management in light of its predecessors.
Go developers pride themselves in three things:
- We are purists, in the good sense. We value elegance over easiness, and also over thoroughness.
- We are practical. YAGNI, KISS, the Law of Demeter, and all that jazz.
- We have learned from the best. We pride ourselves in being able to trace Go's lineage through some of the best languages, operating systems, and theories.
When it comes to package management, we need to stay true to these. And so we start with that great paragon of package management...
Perl's Great Success
Perl was one of the first languages I learned. True to it's namesake, it was practical. But I never particularly enjoyed the language itself. Larry Wall once claimed that Perl was the first postmodern language. I'm not a postmodernist.
Yet Perl absolutely nailed something that made it the towering success it once was: Package management. CPAN was, and remains, one of the greatest package management solutions in the programming language landscape. To what do we attribute its success?
- Using CPAN is easy.
- I can search for packages based on what I want. Typing "json" retrieves a list of suitable packages with useful descriptions.
- It is easy to automate, had a web client, and had a command line client.
- The information most important to developerss (description, license, maintainers, versions, and dependencies) is easy to find.
- As a maintainer of code, it's easy to maintain and publish pertinent metadata
(usually in
Maikefile.PL) - There aren't really any serious "competitors". CPAN has emerged as the standard for distributing modules.
Perl solved this one problem remarkably well. Insodoing, the community transformed a simple reporting language into a powerful platform that led early web application development.
Where Perl Succeeded, Others Failed... Then Succeeded
Java
At around the same time, along came another language: Java. Java was built for the enterprise. But not with the sort of open source landscape that Perl's team had in mind. Java had no dependency management features to speak of. Nor did it have any package management tools. And finding someone else's libraries? That was left as an exercise to the diligent web searcher.
It took the Java community-at-large over a decade to develop a tool that did a decent job of solving these problems. And it is no surprise that Maven skyrocketed in popularity. It was a tool that filled an acute need.
PHP
Perl's biggest challenger for web development came in the form of a little Apache module: PHP. PHP lowered the bar of web development. It won out with web hosting providers and novice web developers alike. And in it's two decades of life, it has gained massive market share. But for many years, it suffered from really horrible (read: nonexistent) package management. Developers simply shared tarballs of code. Or, more often, they didn't share. And the same old bits of logic were rewritten (often poorly) over and over again.
The two results of PHP's packaging failure were:
- Monolithic projects like Joomla, Wordpress, and Drupal gained popularity not just from providing popular features, but because they became hubs for sharing code.
- "Serious" developers mocked PHP for its fractured landscape littered with poorly written projects. So many were poorly written simply because there was little (properly) shared code.
Along the way, PHP tried to replicate Perl's success with Pear, Pecl, and Phar. But each of these suffered from deficiencies that kept package management from succeeding. In no small part, the failures of these technologies had mostly to do with the difficulty in building the packages (and then subsequently finding them). The community unfortunately learned the hard way that poor package management leads to poor code and poor selection.
Decades after its birth, a group of PHP developers came up with a solution. These days, Composer (inspired by NPM and Ruby Gems+Bundles) is the package manager of choice.
But Wait, There're More...
Similar stories of neglectful thinking, poor technical decisions, and unrealistic barriers of entry have made package management a pain point not just for Java and PHP, but also for Ruby, JavaScript, and Python. But each has eventually distilled the list to a few solid package managers. Python has pip. Ruby has Gems and Bundler. Javascript has NPM.
Each of these langauges have eventually made their way to success in the area of packaging. They all seem to have converged upon patterns that are remarkably similar to Perl's CPAN (and also the sort of package management we see in Debain, Fedora, Ubuntu, Gentoo, Arch, and other successful OS package managers).
The repeatable characteristics seem to be:
- Richer metadata about packages, including
- Clear licensing information
- Clear maintainership information
- Clear pointers to documentation
- Standards about what versions mean, and how versioning is resolved
- Paradigms for fetching packages (often over HTTP), and then installing them
to specific locations.
- Many even distinguish between local (app specifig) and global (shared across all projects) installs.
- The "repeatability of the build" is a common theme for fetching and building
- Formats easily consumable by tools, but also easily maintained by humans
- Great tooling
They may differ between centralized and de-centralized repositories, file formats, and so on. But the core items above seem to be tremendously important to managers from dpkg to Maven to CPAN.
Which Route Will Go Go?
Go began with rudimentary package management. But thanks to the successes of the community and the availability of open source Go libraries, it is becoming necessary for Go to provide a better solution to package management. How do we get there?
I started by pointing out three traits that I think define Gophers: Purist, practical, and intent on learning from the best. And in the realm of package management, that seems to lead to a pretty simple conclusion:
Go needs a standard metadata specification that covers the basics, is easy to maintain, and is easy to consume.
And there is a second conclusion we might learn, too: Different is not ipso facto better. When we see convergence across communities and technologies, we should not ignore it and simply assert that we've got some good ideas of our own. (Or, worse, we don't need all that stuff because my current use case doesn't require it.)
It is a good idea to understand why a CPAN file, dpkg manifests, and an NPM file all have remarkably similar structure and contents. There's a lot to learn there. And we're smart... we don't need to repeat the wasted cycles of Java, PHP, Python, and Ruby.
