Why Brigade Doesn't Do YAML
Brigade newcomers often ask me why Brigade doesn't use YAML. Coming from CircleCI, TravisCI, and Kubernetes itself, some developers expect that Brigade should be a "declarative" method of describing a sequence of steps to be run. This is not what Brigade is supposed to be. Brigade is a script execution environment. Brigade is shell scripting for Kubernetes.
This post explains the rationale of Brigade's scripting.
This is the first in a series about Brigade concepts. Current Brigade version: 0.7.
An Operating System Shell Script
Many operating systems provide shell environments in which a programmer can execute a command. The environment provides a context in which programmers can pass information into the executable, and the executable can send information back. Shell scripts provide a mechanism for chaining together multiple commands, and also controlling the execution flow of those commands.
Thus we can express things like this:
#!/bin/bash
for i in $( ls -1 ); do
if [[ -d $i ]]; then
echo "$i is a directory"
fi
done
The above bash script executes the ls command, loops through each path it returns, and checks to see whether that path points to a directory. If the path is a directory, the script echos a confirmation message.
It is important to note that the shell script gains its power from the fact that it is merely providing flow control around a bunch of individual programs on the system.
This is the experience we wanted to provide with Brigade. Except that instead of scripting the flow of programs on a single operating system, we wanted to script containers on a cluster.
A Cluster "Shell Script"
Kubernetes can be seen as a sort of operating system that spans multiple host systems. Instead of executing programs as processes, it executes containers. And just as modern operating systems can schedule processes to run on different cores, Kubernetes can schedule containers to run on different nodes. (The analogy is not exactly one-to-one, but the general idea holds.)
We conceived of Brigade by starting with that analogy, and then asking "so what does shell scripting look like on such an operating system?" The answer followed naturally: Cluster scripting needs to be able to provide flow control that wraps the execution of containers, but is not opinionated about what those containers do or how they run.
Drawing from our earlier shell scripting example, Brigade was initially conceived as providing an experience something like this:
for i in $( run_some_container ); do
if $( run_some_other_container ); then
$( run_yet_another_container )
fi
done
We were confronted with a few realities, though:
- Writing a new language was both unnecessary and impractical (and hard to convince people to use)
- Containers are functionally different from UNIX programs in a few notable ways, and we should preserve the differences, not hide them
- Invoking commands in a cluster might work differently than running local commands
Based on these three discoveries, we made some decisions that are apparent in today's Brigade.
Choose a language, don't write a language
In answer to (1) above, we decided to simply choose an existing scripting language. Personally, I wanted to use Lua. But when we looked at usage patterns and existing developer skills, it was strikingly clear that JavaScript enjoys the broadest adoption and is also flexible enough.
Within the first few days of our mulling over Brigade, we decided on JavaScript.
Don't pretend containers are something they are not
In answer to (2), we decided to let our analogy of executing programs influence our design, but not dictate it.
When one executes a program, the shell provides a number of ways to send information into that program. There are flags, subcommands, arguments, environment variables, and STDIN. Likewise, there are a few ways for the program to send information back to the environment. It can write information to pseudo-files like STDOUT or STDERR, or it can write information into standard files.
We liked this flexibility, but decided to (a) implement it in a JavaScript native way, and (b) allow container paradigms to dictate what inputs and outputs look like. So we ended up with something like this:
var program = new Job("one", "alpine:3.5")
program.tasks = [ "echo hello", "echo goodbye"]
program.env = { "KEY": "value" }
program.run()
Essentially, the above declares program to be an instance of an alpine:3.5 image. We support a few different ways of sending data into this program. Two are exhibited above: tasks is a list of specific things that should be run inside of the container. env is a map of environment variables.
Unlike UNIX shell scripting, we followed a pattern more natural in other programming languages: We broke apart the steps of declaration (create program), setup (add tasks and env), and execution (call run). But even with this syntactic difference, the spirit of what a shell script does remains evident in this pattern.
Invoke scripts via events
To answer (3), we had to answer the question of what sorts of things Brigade must be good at doing. This led us to move from the model of directly executing Brigade scripts to reacting to triggers in the cluster. Another shell scripting language exemplifies this approach.
Applescript is a peculiar language in some ways. Take this short script, which opens a URL in a Firefox tab:
on firefoxRunning()
tell application "System Events" to (name of processes) contains "firefox"
end firefoxRunning
on run argv
if (firefoxRunning() = false) then
do shell script "open -a Firefox " & (item 1 of argv)
else
tell application "Firefox"
activate
open location item 1 of argv
end tell
end if
end run
Applescript is designed to make it easy for the script to react not just to an explicit invocation of a user, but to the events caused by applications running on the system. Thus, Applescript conceptually revolves around creating (tell) and reacting to (on) events.
As we thought about the way that clusters work, this is how we envisioned Brigade scripts functioning. A brigade script listens for, and reacts to, events. Thus, event handlers are the entry points in Brigade scripts:
events.on("someEvent", ( eventData ) => {
var program = Job("one", "alpine:3.5")
program.tasks = [ "echo hello", "echo goodbye"]
program.env = { "KEY": "value" }
program.run()
})
events.on("someOtherEvent", ( eventData ) => {
// Do something else
})
With this design, one Brigade script can react to multiple cluster events.
Conclusion
When designing Brigade, we began with the notion that Kubernetes is operating system-like. Just as modern operating systems benefit from having scripting languages, so would Kubernetes.
We started with a common scripting language so that programmers have a lower barrier of entry.
While we took cues from UNIX shell scripting, we also tried to stay true to the underlying components. Containers are like programs in some ways, but unlike in other crucial ways. And so we built an API that played to the strengths of containers.
Finally, we introduced the event oriented model because we felt it better captured the use cases for a cluster-oriented scripting language. In this way, Brigade scripts have some affinities with languages like Applescript.
We did not set out to create a pipeline description language. We set out to create a cluster scripting environment. As it turned out, pipeline construction is a good way to accomplish many tasks, and so we built good library support for this. But to think of Brigade as merely a way to declare a sequence of tasks is to misunderstand the design.
Early in the design, we considered providing a YAML syntax for simple pipelines. But this simply didn't make sense. It turned Brigade into a caricature of its intended purpose. And so Brigade remains what we set out to make it: A scripting environment for Kubernetes.
