Functional Testing in a Containerized World
In an age of REST, IoT, and cloud services, well constructed functional tests can exercise substantial portions of the code by simply emulating a client of the service, and then testing the environment. But the real challenge of building good functional tests is creating a clean and reliable test environment that reflects the actual deployment platform. And this is where a couple of popular cloud technologies can provide a huge boon to functional testing. With virtualization (e.g. EC2) and containerization (Docker), developers can create disposable and repeatable testing environments that reflect production.
Software developers have a broad taxonomy of testing. Unit tests exercise individual software components in isolation. Regression tests provide a safeguard against the re-appearance of known issues. And functional tests ensure that an application as a whole is properly functioning. Well written functional tests exercise the application fully, and for that reason they may very well offer the most bang for the buck.
Developers typically focus their effort, however, on unit testing. Perhaps this is because they provide a fine level of granularity, and are often the easiest form of testing to code, unit tests are justly popular. Thanks to movements like TDD (Test Driven Development), most developers feel obligated to write at least some unit tests. And, in fact, medium and large projects commonly run unit tests automatically each time code is checked into version control. Unit tests are good, but they do not (by design) guarantee that the whole application is functioning as intended.
Functional tests, in contrast, enjoy lower adoption rates. This is probably related to the traditional workload ascribed to functional testing. Not only must the developer write tests. There is also an environment to set up. This comes complete with databases, message queues, caches, file systems, and whatever other resources the application relies upon in production. And these environments must be maintained while also being protected from the side-effects of catastrophically failing tests. Because they take so much work to maintain, typically only teams with dedicated QA teams bother with full functional tests.
With the maturation of cloud services, though, a batch of alternatives are now surfacing. The process of setting up and tearing down environments used to be labor intensive, time consuming, and error prone. But cloud services make this process scriptable and fast. Environments are no longer sacred ground to be set up and maintained for long periods of time. They are disposable.
When it comes to running servers in the cloud, there are two major approaches. The first is virtualization. This model is popularized by Amazon’s EC2 and Microsoft Azure. Hefty servers provide a virtualization service which allows entire operating systems to execute inside of this host. In this model, the host system emulates the properties of a hardware platform so that the hosted system believes itself to have its own network card, storage, memory, and so on.
Another popular model has emerged, though. Containerization is similar in many respects to virtualization, but with a notable difference. Instead of running an entire operating system for each container, the host system runs a single kernel, but each hosted system gets its own file system and process control. Because more is shared and less is emulated, containers can be created and torn down rapidly. Docker, CoreOS, HashiCorp, and Century Link Labs have each dived into the containerization world and begun creating sophisticated tooling around containers.
Whether virtualized or containerized, the promise of these tools is that we can now think of operating systems as ephemeral. We can set them up and tear them down as it is convenient for us. And thanks to a wealth of new tools, we can automate this process with meticulous precision.
This is where functional testing becomes exciting. By crafting repeatable builds for our virtual machines or containers, we can bring up a new testing environment programmatically, then execute a full batch of tests, analyze the results, and discard the entire environment. Each time this process is initiated, the environment is pristine. And when a test fails catastrophically and, say, corrupts a database, this won’t impact subsequent test runs. The entire environment is destroyed and later re-created afresh.
The most exciting project illustrating this new approach to functional testing is Drone. With commercial services at drone.io, and the entire code made available as open source, Drone is paving the way to a new breed of testing tools. Poised against the common but clunky Jenkins CI server, Drone uses containerization to create entire environments. Using Docker build files and images, developers can run multiple containers, each performing its part, and create (not simulate, not emulate, but actually create) an entire network of services.
For example, a REST-based web service may have a relational database, a web server, file services, and a message queue. In production, each of these may run on separate intastances. In Drone, a developer creates a main container (likely the one running the web server) along with a number of dependent services, like the database and the message queue. Scripted tests can then test the database migrations, the initialization logic, and the functioning of the service. All of this is automated. And once the tests are complete, the results are reported back to Drone itself, which dutifully tracks test runs. Then the test environment is destroyed.
Because these environments can be set up in minutes or even seconds, testing does not become a lengthy ordeal. Developers can expect feedback soon after each code check-in. And because tools like Drone are easily integrated with version control systems like Git, functional testing becomes an unobtrusive part of the development lifecycle. Developers simply push their code into the source code control and the tests are automatically run.
What makes this approach so exciting? There are several reasons. Running tests in isolation, and in a controlled environment, is highly desirable. Because the environment itself is scripted, the testing team gains programatic control over it, too. An environment like Drone allows developers and QA to focus on writing good functional tests, rather than maintaining an extra set of servers.
Drone is at the avant garde of a new breed of testing tools. It exhibits how we can use cloud technologies to achieve greater control over a detailed test suites, and thus build a stable and easy to maintain functional testing platform. Do not be surprised if this nascent market gains focus in the next year or two. Jenkins and the old-style continuous integration platforms cannot hold a candle to this new breed of solution.
This article was originally for a print publication, but it never made it that far.
