TechnoSophosTechnology. Wise.http://technosophos.com/2019-08-07T13:58:00+00:00Matt ButcherWriting a Kubernetes CRD Controller in Rusthttp://technosophos.com/2019/08/07/writing-a-kubernetes-controller-in-rust.html2019-08-07T13:58:00+00:002019-08-07T14:48:31+00:00Article Author<p>In this post, we'll define a Kubernetes Custom Resource Definition (CRD) and then write a controller (or operator) to manage it -- all in 60 lines of Rust code.</p>
<p>Over the last several months, I have been writing more and more Kubernetes-specific code in Rust. Even though Kubernetes itself was written in Go, I am finding that I can typically write more concise, readable, and stable Kubernetes code in Rust. For example, I recently wrote functionally equivalent CRD controllers in Rust and in Go. The Go version was over 1700 lines long and was loaded with boilerplate and auto-generated code. The Rust version was only 127 lines long. It was much easier to understand and debug... and definitely faster to write. Here, we'll write one in just 60 lines.</p>
<h2>Getting Started</h2>
<p>You should have the latest stable Rust release. You'll also need <code>kubectl</code>, configured to point to an existing Kubernetes cluster.</p>
<p>A controller runs as a daemon process, typically inside of a Kubernetes cluster. So we'll create a new Rust program (as opposed to a library). Our aim here is to provide a basic model for writing controllers, so we won't spend time breaking things down into modules. We also won't cover things like building a Rust Docker image or creating a Deployment to run our controller. All of that is well documented elsewhere.</p>
<p>Let's start by creating our new project:</p>
<pre class="highlight shell_session"><code><span class="w">$ </span><span class="nc">cargo</span><span class="kv"> new k8s-controller
</span> Created binary (application) `k8s-controller` package
</code></pre>
<p>Before we start writing code, let's create two YAML files. The first is our CRD definition, and the second is an instance of that CRD. We'll create a directory in <code>k8s-controller/</code> called <code>docs/</code> and put our YAML files there.</p>
<p>The Custom Resource Definition looks like this:</p>
<pre class="highlight yaml"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">apiextensions.k8s.io/v1beta1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">CustomResourceDefinition</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">books.example.technosophos.com</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">group</span><span class="pi">:</span> <span class="s">example.technosophos.com</span>
<span class="na">versions</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">served</span><span class="pi">:</span> <span class="s">true</span>
<span class="na">storage</span><span class="pi">:</span> <span class="s">true</span>
<span class="na">scope</span><span class="pi">:</span> <span class="s">Namespaced</span>
<span class="na">names</span><span class="pi">:</span>
<span class="na">plural</span><span class="pi">:</span> <span class="s">books</span>
<span class="na">singular</span><span class="pi">:</span> <span class="s">book</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Book</span>
</code></pre>
<p>Stepping through this file is beyond the scope of this tutorial, but you can learn all about this file format <a href="https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/">in the official docs</a>. (Recent versions of Kubernetes added more fields to the definition, but we're going to stick with a basic version.) A CRD is just a manifest that declares a new resource type and expresses the names that are associated with this new resource type. The full name of ours is <code>books.example.technosophos.com/v1</code>.</p>
<p>Next, let's make an instance of our <code>Book</code> CRD.</p>
<pre class="highlight yaml"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">example.technosophos.com/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Book</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">moby-dick</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">title</span><span class="pi">:</span> <span class="s">Moby Dick</span>
<span class="na">authors</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">Herman Melville</span>
</code></pre>
<p>As with most Kubernetes resource types, our example above has two main sections:</p>
<ul>
<li><code>metadata</code>, which is a predefined metadata section</li>
<li><code>spec</code>, which holds our custom body</li>
</ul>
<p>We can quickly test to make sure that things are working:</p>
<pre class="highlight shell_session"><code><span class="w">$ </span><span class="nc">kubectl</span><span class="kv"> create -f docs/crd.yaml
</span>customresourcedefinition.apiextensions.k8s.io "books.example.technosophos.com" created
<span class="w">$ </span><span class="nc">kubectl</span><span class="kv"> create -f docs/book.yaml
</span>book.example.technosophos.com "moby-dick" created
<span class="w">$ </span><span class="nc">kubectl</span><span class="kv"> delete book moby-dick
</span>book.example.technosophos.com "moby-dick" deleted
</code></pre>
<p>We now have everything we need to start coding our new controller.</p>
<h3>Setting up our <code>Cargo.toml</code> file</h3>
<p>Rather than incrementally adding dependencies to our <code>Cargo.toml</code> file as we go, we'll just set up all of the dependencies now. As the text progresses, we'll see how these are used.</p>
<pre class="highlight toml"><code><span class="nn">[package]</span>
<span class="py">name</span> <span class="p">=</span> <span class="s">"k8s-controller"</span>
<span class="py">version</span> <span class="p">=</span> <span class="s">"0.1.0"</span>
<span class="py">edition</span> <span class="p">=</span> <span class="s">"2018"</span>
<span class="nn">[dependencies]</span>
<span class="py">kube</span> <span class="p">=</span> <span class="s">"0.14.0"</span>
<span class="py">serde</span> <span class="p">=</span> <span class="s">"1.0"</span>
<span class="py">serde_derive</span> <span class="p">=</span> <span class="s">"1.0"</span>
<span class="py">serde_json</span> <span class="p">=</span> <span class="s">"1.0"</span>
</code></pre>
<p>The <code>serde</code> serialization libraries are likely already familiar to you. And <code>kube</code> is the Kubernetes library for writing controllers. (Another library, <code>k8s_openapi</code>, is useful for working with existing Kubernetes resource types, but we don't need it here)</p>
<h2>Part 1: Create the Book Struct</h2>
<p>The first piece of code we'll write is a struct that represents our book CRD. And the easiest way to start with that is to write the basic struct that defines the body (<code>spec</code>). In our <code>book.yaml</code> we had two fields in <code>spec</code>:</p>
<ul>
<li><code>title</code>: The book's title</li>
<li><code>authors</code>: A list of authors</li>
</ul>
<p>Since we're just writing a quick example, we'll go ahead and create this struct inside of <code>main.rs</code>:</p>
<pre class="highlight rust"><code><span class="err">#</span><span class="p">[</span><span class="n">macro_use</span><span class="p">]</span>
<span class="k">extern</span> <span class="n">crate</span> <span class="n">serde_derive</span><span class="p">;</span>
<span class="c">// This is our new Book struct</span>
<span class="cp">#[derive(Serialize,</span> <span class="cp">Deserialize,</span> <span class="cp">Clone,</span> <span class="cp">Debug)]</span>
<span class="k">pub</span> <span class="k">struct</span> <span class="n">Book</span> <span class="p">{</span>
<span class="k">pub</span> <span class="n">title</span><span class="p">:</span> <span class="nb">String</span><span class="p">,</span>
<span class="k">pub</span> <span class="n">authors</span><span class="p">:</span> <span class="nb">Option</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="nb">String</span><span class="o">>></span><span class="p">,</span>
<span class="p">}</span>
<span class="c">// This was the boilerplate that Cargo generated:</span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"Hello, world!"</span><span class="p">);</span>
<span class="p">}</span>
</code></pre>
<p>By making the <code>title</code> a string and the <code>authors</code> an <code>Option</code>, we're stating that the title is required, but the authors are not. So now we have:</p>
<ul>
<li>A <code>title</code> string</li>
<li>An optional vector of authors as strings</li>
</ul>
<p>We've also used macros to generate the Serde serializer and deserializer features as well as clone and debug support.</p>
<p>If we look again at our <code>book.yaml</code>, we will see that the body of the book has two sections:</p>
<ul>
<li><code>metadata</code> with the name</li>
<li><code>spec</code> with the rest of the data</li>
</ul>
<p>Some Kubernetes objects have a third section called <code>status</code>. We don't need one of those.</p>
<p>The <code>kube</code> library is aware of this <code>metadata</code>/<code>spec</code>/<code>status</code> pattern. So it provides a generic type called <code>kube::api::Object</code> that we can use to create a Kubernetes-style resource. To make our code easier to read, we'll create a type alias for this new resource type:</p>
<pre class="highlight rust"><code><span class="c">// Describes a Kubernetes object with a Book spec and no status</span>
<span class="k">type</span> <span class="n">KubeBook</span> <span class="o">=</span> <span class="n">Object</span><span class="o"><</span><span class="n">Book</span><span class="p">,</span> <span class="n">Void</span><span class="o">></span><span class="p">;</span>
</code></pre>
<p>A <code>cube::api::Object</code> already has the <code>metadata</code> section defined. But it gives us the option of adding our own <code>spec</code> and <code>status</code> fields. We add <code>Book</code> as the spec, but we don't need a status field, so we set it to <code>Void</code>.</p>
<p>Here's the code so far:</p>
<pre class="highlight rust"><code><span class="err">#</span><span class="p">[</span><span class="n">macro_use</span><span class="p">]</span>
<span class="k">extern</span> <span class="n">crate</span> <span class="n">serde_derive</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">kube</span><span class="p">::</span><span class="nn">api</span><span class="p">::{</span><span class="n">Object</span><span class="p">,</span> <span class="n">Void</span><span class="p">};</span>
<span class="cp">#[derive(Serialize,</span> <span class="cp">Deserialize,</span> <span class="cp">Clone,</span> <span class="cp">Debug)]</span>
<span class="k">pub</span> <span class="k">struct</span> <span class="n">Book</span> <span class="p">{</span>
<span class="k">pub</span> <span class="n">title</span><span class="p">:</span> <span class="nb">String</span><span class="p">,</span>
<span class="k">pub</span> <span class="n">authors</span><span class="p">:</span> <span class="nb">Option</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="nb">String</span><span class="o">>></span><span class="p">,</span>
<span class="p">}</span>
<span class="c">// This is a convenience alias that describes the object we get from Kubernetes</span>
<span class="k">type</span> <span class="n">KubeBook</span> <span class="o">=</span> <span class="n">Object</span><span class="o"><</span><span class="n">Book</span><span class="p">,</span> <span class="n">Void</span><span class="o">></span><span class="p">;</span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"Hello, world!"</span><span class="p">);</span>
<span class="p">}</span>
</code></pre>
<p>Now we're ready to work on <code>main()</code>.</p>
<h2>Part 2: Connecting to Kubernetes</h2>
<p>Next, we'll create the controller in the <code>main()</code> function. We'll take this in a few steps. First, let's load all of the information we need in order to work with Kubernetes.</p>
<pre class="highlight rust"><code><span class="err">#</span><span class="p">[</span><span class="n">macro_use</span><span class="p">]</span>
<span class="k">extern</span> <span class="n">crate</span> <span class="n">serde_derive</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">kube</span><span class="p">::{</span>
<span class="nn">api</span><span class="p">::{</span><span class="n">Object</span><span class="p">,</span> <span class="n">Void</span><span class="p">,</span> <span class="n">RawApi</span><span class="p">},</span>
<span class="nn">client</span><span class="p">::</span><span class="n">APIClient</span><span class="p">,</span>
<span class="n">config</span><span class="p">,</span>
<span class="p">};</span>
<span class="cp">#[derive(Serialize,</span> <span class="cp">Deserialize,</span> <span class="cp">Clone,</span> <span class="cp">Debug)]</span>
<span class="k">pub</span> <span class="k">struct</span> <span class="n">Book</span> <span class="p">{</span>
<span class="k">pub</span> <span class="n">title</span><span class="p">:</span> <span class="nb">String</span><span class="p">,</span>
<span class="k">pub</span> <span class="n">authors</span><span class="p">:</span> <span class="nb">Option</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="nb">String</span><span class="o">>></span><span class="p">,</span>
<span class="p">}</span>
<span class="c">// This is a convenience alias that describes the object we get from Kubernetes</span>
<span class="k">type</span> <span class="n">KubeBook</span> <span class="o">=</span> <span class="n">Object</span><span class="o"><</span><span class="n">Book</span><span class="p">,</span> <span class="n">Void</span><span class="o">></span><span class="p">;</span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
<span class="c">// Load the kubeconfig file.</span>
<span class="k">let</span> <span class="n">kubeconfig</span> <span class="o">=</span> <span class="nn">config</span><span class="p">::</span><span class="nf">load_kube_config</span><span class="p">()</span><span class="nf">.expect</span><span class="p">(</span><span class="s">"kubeconfig failed to load"</span><span class="p">);</span>
<span class="c">// Create a new client</span>
<span class="k">let</span> <span class="n">client</span> <span class="o">=</span> <span class="nn">APIClient</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="n">kubeconfig</span><span class="p">);</span>
<span class="c">// Set a namespace. We're just hard-coding for now.</span>
<span class="k">let</span> <span class="n">namespace</span> <span class="o">=</span> <span class="s">"default"</span><span class="p">;</span>
<span class="c">// Describe the CRD we're working with.</span>
<span class="c">// This is basically the fields from our CRD definition.</span>
<span class="k">let</span> <span class="n">resource</span> <span class="o">=</span> <span class="nn">RawApi</span><span class="p">::</span><span class="nf">customResource</span><span class="p">(</span><span class="s">"books"</span><span class="p">)</span>
<span class="nf">.group</span><span class="p">(</span><span class="s">"example.technosophos.com"</span><span class="p">)</span>
<span class="nf">.within</span><span class="p">(</span><span class="o">&</span><span class="n">namespace</span><span class="p">);</span>
<span class="p">}</span>
</code></pre>
<p>If we run this program it won't do anything visible. But here's what's happening in the <code>main()</code> function:</p>
<ul>
<li>First we load the <code>kubeconfig</code> file (or, in cluster, read the secrets out of the volume mounts). This loads the URL to the Kubernetes API server, and also the credentials for authenticating.</li>
<li>Second, we create a new API client. This is the object that will communicate with the Kubernetes API server.</li>
<li>Third, we set the namespace. Kubernetes segments objects by namespaces. In a normal program, we'd provide a way for the user to specify a particular namespace. But for this, we'll just use the <code>default</code> built-in namespace.</li>
<li>Forth, we are creating a <code>resource</code> that describes our CRD. We'll use this in a bit to tell the informer which things it should watch for.</li>
</ul>
<p>So now we have sufficient information to run operations against the Kubernetes API server for our particular namespace and watch for our particular CRD. </p>
<p>Next, we can create an informer.</p>
<h2>Part 3: Creating an Informer</h2>
<p>In Kubernetes parlance, an <em>informer</em> is a special kind of agent that watches the Kubernetes event stream and <em>informs</em> the program when a particular kind of resource triggers an event. This is the heart of our controller.</p>
<blockquote>
<p>There is a second kind of watching agent that keeps a local cache of all objects that match a type. That is called a <em>reflector</em>.</p>
</blockquote>
<p>In our case, we're going to write an informer that tells us any time anything happens to a <code>Book</code>.</p>
<p>Here's the code to create an informer and then handle events as they come in:</p>
<pre class="highlight rust"><code><span class="err">#</span><span class="p">[</span><span class="n">macro_use</span><span class="p">]</span>
<span class="k">extern</span> <span class="n">crate</span> <span class="n">serde_derive</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">kube</span><span class="p">::{</span>
<span class="nn">api</span><span class="p">::{</span><span class="n">Object</span><span class="p">,</span> <span class="n">RawApi</span><span class="p">,</span> <span class="n">Informer</span><span class="p">,</span> <span class="n">WatchEvent</span><span class="p">,</span> <span class="n">Void</span><span class="p">},</span>
<span class="nn">client</span><span class="p">::</span><span class="n">APIClient</span><span class="p">,</span>
<span class="n">config</span><span class="p">,</span>
<span class="p">};</span>
<span class="cp">#[derive(Serialize,</span> <span class="cp">Deserialize,</span> <span class="cp">Clone,</span> <span class="cp">Debug)]</span>
<span class="k">pub</span> <span class="k">struct</span> <span class="n">Book</span> <span class="p">{</span>
<span class="k">pub</span> <span class="n">title</span><span class="p">:</span> <span class="nb">String</span><span class="p">,</span>
<span class="k">pub</span> <span class="n">authors</span><span class="p">:</span> <span class="nb">Option</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="nb">String</span><span class="o">>></span><span class="p">,</span>
<span class="p">}</span>
<span class="c">// This is a convenience alias that describes the object we get from Kubernetes</span>
<span class="k">type</span> <span class="n">KubeBook</span> <span class="o">=</span> <span class="n">Object</span><span class="o"><</span><span class="n">Book</span><span class="p">,</span> <span class="n">Void</span><span class="o">></span><span class="p">;</span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
<span class="c">// Load the kubeconfig file.</span>
<span class="k">let</span> <span class="n">kubeconfig</span> <span class="o">=</span> <span class="nn">config</span><span class="p">::</span><span class="nf">load_kube_config</span><span class="p">()</span><span class="nf">.expect</span><span class="p">(</span><span class="s">"kubeconfig failed to load"</span><span class="p">);</span>
<span class="c">// Create a new client</span>
<span class="k">let</span> <span class="n">client</span> <span class="o">=</span> <span class="nn">APIClient</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="n">kubeconfig</span><span class="p">);</span>
<span class="c">// Set a namespace. We're just hard-coding for now.</span>
<span class="k">let</span> <span class="n">namespace</span> <span class="o">=</span> <span class="s">"default"</span><span class="p">;</span>
<span class="c">// Describe the CRD we're working with.</span>
<span class="c">// This is basically the fields from our CRD definition.</span>
<span class="k">let</span> <span class="n">resource</span> <span class="o">=</span> <span class="nn">RawApi</span><span class="p">::</span><span class="nf">customResource</span><span class="p">(</span><span class="s">"books"</span><span class="p">)</span>
<span class="nf">.group</span><span class="p">(</span><span class="s">"example.technosophos.com"</span><span class="p">)</span>
<span class="nf">.within</span><span class="p">(</span><span class="o">&</span><span class="n">namespace</span><span class="p">);</span>
<span class="c">// Create our informer and start listening.</span>
<span class="k">let</span> <span class="n">informer</span> <span class="o">=</span> <span class="nn">Informer</span><span class="p">::</span><span class="nf">raw</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="n">resource</span><span class="p">)</span><span class="nf">.init</span><span class="p">()</span><span class="nf">.expect</span><span class="p">(</span><span class="s">"informer init failed"</span><span class="p">);</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="n">informer</span><span class="nf">.poll</span><span class="p">()</span><span class="nf">.expect</span><span class="p">(</span><span class="s">"informer poll failed"</span><span class="p">);</span>
<span class="c">// Now we just do something each time a new book event is triggered.</span>
<span class="k">while</span> <span class="k">let</span> <span class="nf">Some</span><span class="p">(</span><span class="n">event</span><span class="p">)</span> <span class="o">=</span> <span class="n">informer</span><span class="nf">.pop</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">handle</span><span class="p">(</span><span class="n">event</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">handle</span><span class="p">(</span><span class="n">event</span><span class="p">:</span> <span class="n">WatchEvent</span><span class="o"><</span><span class="n">KubeBook</span><span class="o">></span><span class="p">)</span> <span class="p">{</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"Something happened to a book"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre>
<p>In the code above, we've added a new informer:</p>
<pre class="highlight rust"><code><span class="k">let</span> <span class="n">informer</span> <span class="o">=</span> <span class="nn">Informer</span><span class="p">::</span><span class="nf">raw</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="n">resource</span><span class="p">)</span><span class="nf">.init</span><span class="p">()</span><span class="nf">.expect</span><span class="p">(</span><span class="s">"informer init failed"</span><span class="p">);</span>
</code></pre>
<p>This line creates a <em>raw</em> informer. A raw informer is one that does not use the Kubernetes OpenAPI spec to decode its contents. Since we are using a custom CRD, we don't need the OpenAPI spec. Note that we give this informer two pieces of information:</p>
<ul>
<li>A Kubernetes client that can talk to the API server</li>
<li>The <code>resource</code> that tells the informer what we want to watch for</li>
</ul>
<p>Based on these pieces of information, our informer will now connect to the API server and watch for any events having to do with our Book CRD. Next, we just need to tell it to keep listening for new events:</p>
<pre class="highlight rust"><code> <span class="k">loop</span> <span class="p">{</span>
<span class="n">informer</span><span class="nf">.poll</span><span class="p">()</span><span class="nf">.expect</span><span class="p">(</span><span class="s">"informer poll failed"</span><span class="p">);</span>
<span class="c">// Now we just do something each time a new book event is triggered.</span>
<span class="k">while</span> <span class="k">let</span> <span class="nf">Some</span><span class="p">(</span><span class="n">event</span><span class="p">)</span> <span class="o">=</span> <span class="n">informer</span><span class="nf">.pop</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">handle</span><span class="p">(</span><span class="n">event</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
<p>The above tells the informer to poll the API server. Each time a new event is queued, <code>pop()</code> takes the event off of the queue and handles it. Right now, our <code>handle()</code> method is unimpressive:</p>
<pre class="highlight rust"><code><span class="k">fn</span> <span class="nf">handle</span><span class="p">(</span><span class="n">event</span><span class="p">:</span> <span class="n">WatchEvent</span><span class="o"><</span><span class="n">KubeBook</span><span class="o">></span><span class="p">)</span> <span class="p">{</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"Something happened to a book"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre>
<p>In a moment, we'll add some features to <code>handle()</code>, but first let's see what happens if we run this code.</p>
<p>In one terminal, start <code>cargo run</code> and leave it running.</p>
<pre class="highlight shell_session"><code><span class="w">$ </span><span class="nc">cargo</span><span class="kv"> run
</span> Finished dev [unoptimized + debuginfo] target(s) in 7.28s
Running `target/debug/k8s-controller`
</code></pre>
<blockquote>
<p>Make sure your local environment is pointed to a Kubernetes cluster! Otherwise neither <code>cargo run</code> nor the <code>kubectl</code> commands will work. And make sure you installed <code>docs/crd.yaml</code>.</p>
</blockquote>
<p>Now, with that running in one terminal, we can run this in another:</p>
<pre class="highlight shell_session"><code><span class="w">$ </span><span class="nc">kubectl</span><span class="kv"> create -f docs/book.yaml
</span><span class="ni"># </span><span class="nc">wait</span><span class="kv"> for a bit
</span><span class="w">$ </span><span class="nc">kubectl</span><span class="kv"> delete book moby-dick
</span></code></pre>
<p>In the <code>cargo run</code> console, we'll see this:</p>
<pre class="highlight plaintext"><code> Finished dev [unoptimized + debuginfo] target(s) in 7.28s
Running `target/debug/k8s-controller`
Something happened to a book
Something happened to a book
</code></pre>
<p>In the final section, we'll add a little more to the <code>handle()</code> function.</p>
<h2>Part 4: Handling Events</h2>
<p>In this last part, we'll add a few more things to the <code>handle()</code> function. Here is our revised function:</p>
<pre class="highlight rust"><code><span class="k">fn</span> <span class="nf">handle</span><span class="p">(</span><span class="n">event</span><span class="p">:</span> <span class="n">WatchEvent</span><span class="o"><</span><span class="n">KubeBook</span><span class="o">></span><span class="p">)</span> <span class="p">{</span>
<span class="c">// This will receive events each time something </span>
<span class="k">match</span> <span class="n">event</span> <span class="p">{</span>
<span class="nn">WatchEvent</span><span class="p">::</span><span class="nf">Added</span><span class="p">(</span><span class="n">book</span><span class="p">)</span> <span class="k">=></span> <span class="p">{</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"Added a book {} with title '{}'"</span><span class="p">,</span> <span class="n">book</span><span class="py">.metadata.name</span><span class="p">,</span> <span class="n">book</span><span class="py">.spec.title</span><span class="p">)</span>
<span class="p">},</span>
<span class="nn">WatchEvent</span><span class="p">::</span><span class="nf">Deleted</span><span class="p">(</span><span class="n">book</span><span class="p">)</span> <span class="k">=></span> <span class="p">{</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"Deleted a book {}"</span><span class="p">,</span> <span class="n">book</span><span class="py">.metadata.name</span><span class="p">)</span>
<span class="p">}</span>
<span class="n">_</span> <span class="k">=></span> <span class="p">{</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"another event"</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
<p>Note that the function signature says that it accepts <code>event: WatchEvent<KubeBook></code>. The informer emits <code>WatchEvent</code> objects that describe the event that it saw occur on the Kubernetes event stream. When we created the informer, we told it to watch for a <code>resource</code> that described our Book CRD.</p>
<p>So each time a <code>WatchEvent</code> is emitted, it will wrap a <code>KubeBook</code> object. And that object will represent our earlier YAML definition:</p>
<pre class="highlight yaml"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">example.technosophos.com/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Book</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">moby-dick</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">title</span><span class="pi">:</span> <span class="s">Moby Dick</span>
<span class="na">authors</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">Herman Melville</span>
</code></pre>
<p>So we would expect that a <code>KubeBook</code> would have fields like <code>book.metadata.name</code> or <code>book.spec.title</code>. In fact, all of the attributes of our earlier <code>Book</code> struct will be available on the <code>book.spec</code>.</p>
<p>There are four possible <code>WatchEvent</code> events:</p>
<ul>
<li><code>WatchEvent::Added</code>: A new book CRD instance was created</li>
<li><code>WatchEvent::Deleted</code>: An existing book instance was deleted</li>
<li><code>WatchEvent::Modified</code>: An existing book instance was changed</li>
<li><code>WatchEvent::Error</code>: An error having to do with the book watcher occurred</li>
</ul>
<p>In our code above, we use a <code>match event</code> to match on one of the events. We explicitly handle <code>Added</code> and <code>Deleted</code>, but capture the others with the generic <code>_</code> match.</p>
<p>To look closer, in the first match we simply print out the book object's name and the book's title:</p>
<pre class="highlight rust"><code><span class="nn">WatchEvent</span><span class="p">::</span><span class="nf">Added</span><span class="p">(</span><span class="n">book</span><span class="p">)</span> <span class="k">=></span> <span class="p">{</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"Added a book {} with title '{}'"</span><span class="p">,</span> <span class="n">book</span><span class="py">.metadata.name</span><span class="p">,</span> <span class="n">book</span><span class="py">.spec.title</span><span class="p">)</span>
<span class="p">},</span>
</code></pre>
<p>If we execute <code>cargo run</code> and then run our <code>kubectl create</code> and <code>kubectl delete</code> commands again, this is what we'll see in the <code>cargo run</code> output:</p>
<pre class="highlight plaintext"><code>$ cargo run
Compiling k8s-controller v0.1.0 (/Users/technosophos/Code/Rust/k8s-controller)
Finished dev [unoptimized + debuginfo] target(s) in 5.33s
Running `target/debug/k8s-controller`
Added a book moby-dick with title 'Moby Dick'
Deleted a book moby-dick
</code></pre>
<p>From here, we might want to do something more sophisticated with our informer. Or we might want to instead experiment with a reflector. But in just 60 lines of code we have written an entire Kubernetes controller with a Custom Resource Definition!</p>
<h2>Conclusion</h2>
<p>That is all there is to creating a basic controller. Unlike writing these in Go, you won't need special code generators or annotations, gobs of boilerplate code, and complex configurations. This is a fast and efficient way of creating new Kubernetes controllers.</p>
<p>From here, you may want to take a closer look at the <a href="https://crates.io/crates/kube">kube library's documentation</a>. There are dozens of examples, and the API itself is well documented. You will also learn how to work with built-in Kubernetes types (also an easy thing to do).</p>
<p>The code for this post is available at <a href="https://github.com/technosophos/rust-k8s-controller/">github.com/technosophos/rust-k8s-controller</a></p>
<p>You can find the final code <a href="https://github.com/technosophos/rust-k8s-controller/blob/master/src/main.rs">in the GitHub copy of main.rs</a></p>
The TRUE hardest programming problem is tight vs. weak couplinghttp://technosophos.com/2018/08/19/the-true-hardest-problem-is-tight-vs-weak-coupling.html2018-08-19T22:16:00+00:002018-08-20T01:45:25+00:00Article Author<p>A few months ago, I claimed that <a href="/2018/03/08/why-naming-is-the-hardest-programming-problem.html">naming is the hardest programming problem</a>. I was wrong. The true hardest problem is one that impacts every developer at every skill level, across all programming languages, regardless of experience. It appears on multiple levels, from language details to large scale distributed computing. It is equally applicable across all programming disciplines. And its impacts are monetary, hedonic, and cognitive.</p>
<p>The hardest problem in programming is this:</p>
<blockquote>
<p>Should X and Y be tightly coupled or weakly coupled?</p>
</blockquote>
<p>I was trained as a philosopher. Philosophers are a motley bunch, but if there is any generalization that is fair to level at all philosophers, it's this: Philosophers revel in questions that appear to have easy answers, but which, upon reflection, might just be intractable. This appears to be one of those.</p>
<h2>What Do We Mean by Tight and Weak Coupling?</h2>
<p>When we talk about <em>coupling</em>, we're talking about establishing a relationship between two things. While programming, we might couple two functions like this:</p>
<pre class="highlight javascript"><code><span class="kd">function</span> <span class="nx">x</span><span class="p">()</span> <span class="p">{</span>
<span class="c1">//do something</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">y</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">x</span><span class="p">()</span>
<span class="p">}</span>
</code></pre>
<p>By making <code>y()</code> dependent on <code>x()</code>, we've coupled them. But coupling is a more generic term. We could reverse the dependency (have <code>x()</code> call <code>y()</code>). We could have a mutual dependency (<code>x()</code> calls <code>y()</code>, which calls <code>x()</code>). We could add a layer of indirection, such as have <code>y()</code> call an implementation of interface <code>X</code>, and have <code>x()</code> be an implementation of <code>X</code> that is available to <code>y()</code>, and so on.</p>
<p>This particular domain is especially interesting because it admits of multiple levels and abstractions. We can talk not just about functions being coupled, but other common things:</p>
<ul>
<li>Libraries can be coupled</li>
<li>Programs can be coupled (program <code>x</code> calls program <code>y</code>...)</li>
<li>Services can be coupled (which is the basis for microservice architecture)</li>
<li>Data formats can be coupled (such as of SGML, HTML, XML, SVG, and so on)</li>
<li>Protocols can be coupled</li>
<li>User interface elements can be coupled</li>
<li>Hardware and software can be coupled</li>
<li>Network stacks can be coupled</li>
<li>We can even get into high level techs like websites, phones, relations between datacenters, and so on.</li>
</ul>
<p>And to really drive the point home, the new FaaS (Functions as a Service) paradigm is all about coupling "functions" that are each stand-alone services. The hype generated around this technology revolves around this idea that FaaS provides a desirable boundary across which "functions" couple.</p>
<p>So part of what makes this problem interesting is that it is likely be hit by everyone from CS101 students to datacenter operators to architects to data scientists. Everyone in our space deals with coupling, and anyone making design decisions will have to make decisions about coupling.</p>
<p>Next, we need to distinguish between two different kinds of coupling, which many computer scientists refer to as tight and weak coupling.</p>
<h3>Tight Coupling</h3>
<p>It is easier to start with the more restrictive case, and that is tight coupling.</p>
<blockquote>
<p>When establishing a relationship from X to Y, only the mutual needs of X and Y are considered.</p>
</blockquote>
<p>What this says is that tight coupling consists in reducing a coupling problem to only two questions: What does X need to be successfully related to Y? And what does Y need to be successfully related to X?</p>
<p>In programming, this might play out as deciding how to break up programming logic into different functions, and then make calls from one function to another. At a higher level, two microservices are tightly coupled when one service's sole role is to fulfill the needs of one other service.</p>
<p>There's actually a very interesting question to ask at this point: <em>Is coupling about intention or about reality?</em> That is, when we say X is tightly coupled to Y, do we mean "X was designed in such a way as to be related to Y in an exclusive way?" Or do we mean, "X happens to be related to Y in an exclusive way"? It might be easiest to explain that in light of the microservice example above:</p>
<p><em>Option A:</em> My microservice Y was <em>designed</em> only to interoperate with microservice X.</p>
<p><em>Option B:</em> As it happens, microservice Y only interoperates with microservice X.</p>
<p>Clearly, there are many cases where Option B arises in the wild. But those cases are not particularly... how should we say it... <em>philosophically interesting</em>. There are no engaging problems to solve.</p>
<p>But when it comes to <em>intentions</em>, then we are onto something interesting. For in this case we can ask what <em>ought</em> to be done, and how we <em>should</em> make decisions.</p>
<p>At this point, we are talking about making design decisions that involve coupling X to Y by considering only the needs of X and Y.</p>
<h3>Loose Coupling</h3>
<p>Defining loose coupling now no appears to be a boring exercise:</p>
<blockquote>
<p>It is not the case that when establishing a relationship from X to Y, only the mutual needs of X and Y are considered.</p>
</blockquote>
<p>But we can really ignore a bunch of logical cases we don't care about (in which we're focusing on the antecedent) do a little rewording, and give an account of loose coupling that is not a negation of strong coupling, but does provide a <em>relevant alternative</em> to string coupling:</p>
<blockquote>
<p>When establishing a relationship between X and Y, the mutual needs of X and Y are considered along with additional needs.</p>
</blockquote>
<p>But "additional needs" is frustratingly vague.</p>
<blockquote>
<p>Given a system of abstraction S, composed of individual component parts, when establishing the relationship between X (a component of S) and Y (a component of S), the needs of each component in S are considered <em>as they pertain to the relationship between X and Y</em>.</p>
</blockquote>
<p>Note that since X and Y are both components of S, their needs are each considered. But have we just kicked the ambiguity up a level? Because now we are talking about components <em>as they pertain to...</em>. To get rid of this is going to be tedious.</p>
<blockquote>
<p>Given a system of abstraction S, composed of individual component parts, and where X and Y are each component parts, when establishing a relationship from X to Y, the needs of each component's relationship to Y, and the needs of X's relationship to each component must be considered.</p>
</blockquote>
<p>The problem with this definition is that we don't need to consider <em>every possible</em> relationship. That is, when looking at how function X calls function Y to sum a few numbers, we shouldn't also have to look at how X calls Z to check whether a string contains a substrings. We <em>just</em> care about the particular relationship that is under scrutiny. (e.g. what we are really asking is whether Y should or could be used by other components rather than just X, and (vice versa) whether X should be able to use components other than Y to perform the required summing task?</p>
<p>So now we're onto a seriously frustrating definition:</p>
<blockquote>
<p>Given a system of abstraction S, composed of individual component parts, and where X and Y are each component parts, when establishing a particular relationship R from X to Y, the needs of each component's relationship R to Y, and the needs of X's relationship R to each component must be considered.</p>
</blockquote>
<p>By limiting the definition to just a particular relationship makes things less generic.</p>
<p>At this point, one might argue that we've gotten overzealous. Do we really need to consider <em>each component</em> in the system? The short answer is <em>yes</em>. Really, it's a <em>yes, because...</em>. It is perfectly legitimate to apply broad heuristics and say, "When considering the relationship R, I simply ruled out a whole bunch of components because they weren't directly relatable."</p>
<p>But wait! There's more! We need to pull off a very dangerous philosophical move and leave the realm of the existing system, entering the realm of the <em>possible</em>. Because we also need to say, "what if at some point I write new code that does Z... will it, too, need a relationship R with X?"</p>
<p>(If you're keeping track... we started with predicate logic, worked our way into set logic, and are now in modal logic. This problem is a massive pain in the butt.) So we somewhat need to revise our statement to be thus:</p>
<blockquote>
<p>Given a system of abstraction S, composed of all possible individual component parts, and where X and Y are each possible component parts, when establishing a particular relationship R from X to Y, the needs of each component's relationship R to Y, and the needs of X's relationship R to each possible component must be considered.</p>
</blockquote>
<p>Now the problem we skirted earlier might actually be a <em>real</em> problem. For while we can, with some justification, rule out a broad number of actual cases in our code, the set of <em>possible</em> components is highly likely to be substantially larger. Which means, I'm afraid, that we are going to have to cheat... err... be instrumental.</p>
<blockquote>
<p>Given a system of abstraction S, composed of all possible individual component parts, and where X and Y are each possible component parts, when establishing a particular relationship R from X to Y, the needs of each component's relationship R to Y, and the needs of X's relationship R to each <em>relevant</em> possible component must be considered.</p>
</blockquote>
<p>And now we use "relevance" to give us a cognitive safety net, fleshing it out "within scope of system S at time T (when the decision is being made)" and then declaring "within scope" to include a cognitive boundary. Or, to put it in plain English, "relevant" is shorthand for "stuff that seemed to me to be likely at the time."</p>
<p>Were this a proper philosophy paper, we would now revisit our definition of strong coupling, and would discover that we needed to enfancificate it as well. We'd add our systems wording, and our revised relationship wording, but it would still mean the same thing.</p>
<p>Instead, let's bump our definitions from the realm of set logic back to a simple grokkable definition:</p>
<blockquote>
<p>When establishing a particular relationship between X and Y...
* Strong coupling says we only considered the mutual needs of X and Y
* Weak coupling says we consider the other relevant components in the system as well</p>
</blockquote>
<h2>How is this a Problem?</h2>
<p>We've spent some serious wordcount just trying to explain the terms. But does any of this justify claiming that this distinction is at the heart of the hardest problem in programming?</p>
<p>Let's lay down the problem plainly: The hardest problem in programming is assessing, in any given circumstances, the myriad problems associated with coupling. Here's a two-pronged approach for illustrating just how deeply the problem is. First, I'll pick a particular programming challenge, and show the breadth of issues associated with coupling. Then I'll list out a variety of broader circumstance, each of which will admit a similar breadth of issues. In other words, we're doing something like tracing the perimeter of an issue in order to assess the area of the issue.</p>
<p>XXX</p>
<p>Now we can enumerate the areas in which problems like the above might manifest:</p>
<ul>
<li>Do I write generalized classes (getters and setters for all the things?) or just do what's necessary for now?</li>
<li>Do I make the class/function/interface/variable public or private?</li>
<li>Do I expose this part of the library as part of the public API/SDK or leave it internal?</li>
<li>Do I expose this information on the REST API?</li>
</ul>
<h2>- Do I allow this data to be mutated, or just accessed?</h2>
<p>All of these questions have at their core the question of whether the implementation is designed to tightly couple ("This API is private because only the internals should use it") or loosely couple ("This API is public, and thus I have to design for <em>possible</em> use cases").</p>
From Go To Rust - Advanced Testinghttp://technosophos.com/2018/07/25/from-go-to-rust-advanced-testing.html2018-07-25T01:08:00+00:002018-07-25T04:08:10+00:00Article Author<p>For the fifth installment of this series, we'll take a look at benchmarking, documentation testing, and integration testing. As usual, we'll start with an example in Go and see how it translates to Rust. Along the way, we'll be learning about the Rust language.</p>
<p>If you want to catch up on the series:</p>
<ul>
<li>We started with <a href="/2018/05/27/the-go-developers-quickstart-guide-to-rust.html">the basics of working with Rust</a>, and how that compared to Go.</li>
<li>Then we <a href="/2018/06/04/from-go-to-rust-with-an-http-server.html">focused on web services</a> for the second post.</li>
<li>The third post focused on <a href="/2018/06/12/from-go-to-rust-json-and-yaml.html">working with file formats like JSON and YAML</a></li>
<li>And in the previous post we took a look at <a href="/2018/07/07/from-go-to-rust-testing.html">unit testing</a>.</li>
</ul>
<p>As I compared Go and Rust in the last post, I noted that Go has support for testing beyond mere unit tests. And that is where we'll start today.</p>
<h2>Go Goes Beyond Unit Tests</h2>
<p>Go's built-in <code>testing</code> package defines three classes of tests:</p>
<ul>
<li>Unit tests, which we looked at last week</li>
<li>Benchmarks for testing performance of your code</li>
<li>Documentation functions, which can be tested automatically</li>
</ul>
<p>So let's kick things off with an example of benchmarking and documentation functions. We'll continue where we left off last time. Here's the base library that we are writing tests for.</p>
<p>wordutils.go:</p>
<pre class="highlight go"><code><span class="k">package</span><span class="x"> </span><span class="n">wordutils</span><span class="x">
</span><span class="k">import</span><span class="x"> </span><span class="p">(</span><span class="x">
</span><span class="s">"bufio"</span><span class="x">
</span><span class="s">"strings"</span><span class="x">
</span><span class="p">)</span><span class="x">
</span><span class="c">// Initials returns a string with the first letter of each word in the given string.</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">Initials</span><span class="p">(</span><span class="n">phrase</span><span class="x"> </span><span class="kt">string</span><span class="p">)</span><span class="x"> </span><span class="p">(</span><span class="kt">string</span><span class="p">,</span><span class="x"> </span><span class="kt">error</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">wrds</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">words</span><span class="p">(</span><span class="n">phrase</span><span class="p">)</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="s">""</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">initials</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="s">""</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">_</span><span class="p">,</span><span class="x"> </span><span class="n">word</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="k">range</span><span class="x"> </span><span class="n">wrds</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">initials</span><span class="x"> </span><span class="o">+=</span><span class="x"> </span><span class="n">word</span><span class="p">[</span><span class="m">0</span><span class="o">:</span><span class="m">1</span><span class="p">]</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="n">strings</span><span class="o">.</span><span class="n">ToUpper</span><span class="p">(</span><span class="n">initials</span><span class="p">),</span><span class="x"> </span><span class="no">nil</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">words</span><span class="p">(</span><span class="n">str</span><span class="x"> </span><span class="kt">string</span><span class="p">)</span><span class="x"> </span><span class="p">([]</span><span class="kt">string</span><span class="p">,</span><span class="x"> </span><span class="kt">error</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">wordList</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">{}</span><span class="x">
</span><span class="n">scanner</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">bufio</span><span class="o">.</span><span class="n">NewScanner</span><span class="p">(</span><span class="n">strings</span><span class="o">.</span><span class="n">NewReader</span><span class="p">(</span><span class="n">str</span><span class="p">))</span><span class="x">
</span><span class="n">scanner</span><span class="o">.</span><span class="n">Split</span><span class="p">(</span><span class="n">bufio</span><span class="o">.</span><span class="n">ScanWords</span><span class="p">)</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">scanner</span><span class="o">.</span><span class="n">Scan</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">wordList</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="nb">append</span><span class="p">(</span><span class="n">wordList</span><span class="p">,</span><span class="x"> </span><span class="n">scanner</span><span class="o">.</span><span class="n">Text</span><span class="p">())</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="n">wordList</span><span class="p">,</span><span class="x"> </span><span class="n">scanner</span><span class="o">.</span><span class="n">Err</span><span class="p">()</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre>
<p>I am not going to reproduce the unit tests we covered last week. Instead, we are going to dive right into the new testing material, which will also be in <code>wordutils_test.go</code>.</p>
<pre class="highlight go"><code><span class="k">package</span><span class="x"> </span><span class="n">wordutils</span><span class="x">
</span><span class="k">import</span><span class="x"> </span><span class="p">(</span><span class="x">
</span><span class="s">"fmt"</span><span class="x">
</span><span class="s">"testing"</span><span class="x">
</span><span class="p">)</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">BenchmarkInitials</span><span class="p">(</span><span class="n">b</span><span class="x"> </span><span class="o">*</span><span class="n">testing</span><span class="o">.</span><span class="n">B</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">text</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="s">"I have measured my life in coffee spoons"</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">i</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="m">0</span><span class="p">;</span><span class="x"> </span><span class="n">i</span><span class="x"> </span><span class="o"><</span><span class="x"> </span><span class="n">b</span><span class="o">.</span><span class="n">N</span><span class="p">;</span><span class="x"> </span><span class="n">i</span><span class="o">++</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">_</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">Initials</span><span class="p">(</span><span class="n">text</span><span class="p">);</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="nb">panic</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">ExampleInitials</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">text</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="s">"J. Alfred Prufrock"</span><span class="x">
</span><span class="n">out</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">Initials</span><span class="p">(</span><span class="n">text</span><span class="p">)</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="nb">panic</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">fmt</span><span class="o">.</span><span class="n">Print</span><span class="p">(</span><span class="n">out</span><span class="p">)</span><span class="x">
</span><span class="c">// Output:</span><span class="x">
</span><span class="c">// JAP</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre>
<p>The <code>BenchmarkInitials</code> function will be called when we execute <code>go test -bench</code>. It will run the test multiple times until suitable benchmarks can be generated:</p>
<pre class="highlight shell_session"><code><span class="w">$ </span><span class="nc">go</span><span class="kv"> test --bench .
</span>goos: darwin
goarch: amd64
pkg: github.com/technosophos/wordutils
BenchmarkInitials-4 500000 2286 ns/op
PASS
ok github.com/technosophos/wordutils 1.180s
</code></pre>
<p>The <code>ExampleInitials</code> function is part of the documentation. So if we run <code>godoc</code> on our library, we will see the example: <code>godoc -html github.com/technosophos/wordutils Initials</code>. (Unfortunately, the examples are not printed in the plain-text version of <code>godoc</code> help.)</p>
<p>But to make sure that our examples stay current and accurate, Go will automatically run them as tests during regular unit testing:</p>
<pre class="highlight shell_session"><code><span class="w">$ </span><span class="nc">go</span><span class="kv"> test -v .
</span>=== RUN TestWords
--- PASS: TestWords (0.00s)
=== RUN TestInitials
--- PASS: TestInitials (0.00s)
=== RUN ExampleInitials
--- PASS: ExampleInitials (0.00s)
PASS
ok github.com/technosophos/wordutils (cached)
</code></pre>
<p>As with unit tests, Go determines the kind of test based on the function signature.</p>
<ul>
<li><code>BenchmarkXXX(b *testing.B)</code> is a benchmark</li>
<li><code>ExampleXXX()</code> is an example</li>
</ul>
<p>There are a few other variations of these patterns that you can use, but the basic idea is that the testing tool reflects over the code to determine what to execute during a testing cycle.</p>
<p>As we'll see with Rust, there are four supported classes of tests:</p>
<ul>
<li>Unit tests (again, covered last time)</li>
<li>Benchmarks, which are new and still marked unstable</li>
<li>Documentation examples</li>
<li>Integration tests</li>
</ul>
<h2>Rust Benchmarks</h2>
<p>Rust is introducing benchmark testing. It is available in the unstable builds of Rust, but not yet in the official stable build.</p>
<h3>Enabling Benchmarking</h3>
<p>So to test this out, we need to enable unstable features. Inside of the <code>wordutils</code> project, we need to run <code>rustup override add nightly</code> to switch us over to using the nightly build for this particular project.</p>
<pre class="highlight shell_session"><code><span class="w">$ </span><span class="nc">rustup</span><span class="kv"> update nightly
</span><span class="w">$ </span><span class="nc">rustup</span><span class="kv"> override add nightly
</span>info: using existing install for 'nightly-x86_64-apple-darwin'
info: override toolchain for '/Users/mbutcher/Code/Rust/wordutils' set to 'nightly-x86_64-apple-darwin'
nightly-x86_64-apple-darwin unchanged - rustc 1.29.0-nightly (6a1c0637c 2018-07-23)
</code></pre>
<p>Now we can use the benchmarking features.</p>
<h3>Writing Benchmark Tests</h3>
<p>Last time we created a <code>wordutils</code> library with Cargo. By the end, we were experimenting with several features of package organization. But let's start off with a simplified version of the code we used last time, and add just the benchmark.</p>
<pre class="highlight rust"><code><span class="err">#</span><span class="o">!</span><span class="p">[</span><span class="nf">feature</span><span class="p">(</span><span class="n">test</span><span class="p">)]</span>
<span class="k">extern</span> <span class="n">crate</span> <span class="n">test</span><span class="p">;</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="nf">initials</span><span class="p">(</span><span class="n">phrase</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">)</span> <span class="k">-></span> <span class="nb">String</span> <span class="p">{</span>
<span class="n">phrase</span><span class="nf">.split_whitespace</span><span class="p">()</span><span class="nf">.map</span><span class="p">(</span>
<span class="p">|</span><span class="n">word</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">|</span> <span class="n">word</span><span class="nf">.chars</span><span class="p">()</span><span class="nf">.next</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">()</span>
<span class="p">)</span><span class="py">.collect</span><span class="p">::</span><span class="o"><</span><span class="nb">String</span><span class="o">></span><span class="p">()</span><span class="nf">.to_uppercase</span><span class="p">()</span>
<span class="p">}</span>
<span class="cp">#[cfg(test)]</span>
<span class="k">mod</span> <span class="n">tests</span> <span class="p">{</span>
<span class="k">use</span> <span class="nn">super</span><span class="p">::</span><span class="o">*</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">test</span><span class="p">::</span><span class="n">Bencher</span><span class="p">;</span>
<span class="cp">#[bench]</span>
<span class="k">fn</span> <span class="nf">bench_initials</span><span class="p">(</span><span class="n">b</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="n">Bencher</span><span class="p">)</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">input</span> <span class="o">=</span> <span class="s">"J. Alfred Prufrock"</span><span class="p">;</span>
<span class="n">b</span><span class="nf">.iter</span><span class="p">(||</span> <span class="nf">initials</span><span class="p">(</span><span class="n">input</span><span class="p">));</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
<p>Since we are using an unstable feature, we need to tell Rust explicitly that we know what we're doing when we use the <code>test</code> module:</p>
<pre class="highlight plaintext"><code>#![feature(test)]
extern crate test;
</code></pre>
<p>By enabling the test feature, we indicate that we are using features that would otherwise be disabled because of stability flags.</p>
<blockquote>
<p>The stability mechanism of Rust is a cool way of gradually introducing features, while letting people like us kick the tires and report errors.</p>
</blockquote>
<p>Inside of our own <code>mod test</code>, we add just one benchmarking test:</p>
<pre class="highlight rust"><code><span class="k">use</span> <span class="nn">test</span><span class="p">::</span><span class="n">Bencher</span><span class="p">;</span>
<span class="cp">#[bench]</span>
<span class="k">fn</span> <span class="nf">bench_initials</span><span class="p">(</span><span class="n">b</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="n">Bencher</span><span class="p">)</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">input</span> <span class="o">=</span> <span class="s">"I have measured my life in coffee spoons"</span><span class="p">;</span>
<span class="n">b</span><span class="nf">.iter</span><span class="p">(||</span> <span class="nf">initials</span><span class="p">(</span><span class="n">input</span><span class="p">));</span>
<span class="p">}</span>
</code></pre>
<p>Instead of using the <code>#[test]</code> attribute, we use <code>#[bench]</code> to indicate that this is a benchmark. (Unlike Go, Rust function names have no impact on whether this is considered a benchmark test.)</p>
<p>The <code>Bencher</code> is Rust's equivalent of <code>testing.B</code> in Go. We've seen in previous posts how Rust uses iterators and anonymous functions. And in the example above, Rust is doing basically the same thing that Go does, only more compactly.</p>
<p>In Go, we wrote a benchmark like this:</p>
<pre class="highlight go"><code><span class="n">text</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="s">"I have measured my life in coffee spoons"</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">i</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="m">0</span><span class="p">;</span><span class="x"> </span><span class="n">i</span><span class="x"> </span><span class="o"><</span><span class="x"> </span><span class="n">b</span><span class="o">.</span><span class="n">N</span><span class="p">;</span><span class="x"> </span><span class="n">i</span><span class="o">++</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">_</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">Initials</span><span class="p">(</span><span class="n">text</span><span class="p">);</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="nb">panic</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre>
<p>Looking at the <code>for</code> loop, we can see that we ran the test as many times as <code>b.N</code> indicates. But we did have to explicitly create the <code>for</code> loop for this.</p>
<p>Conceptually, Rust is doing the same thing. It is running the <code>|| initials(input)</code> test as many times as <code>bench.iter()</code> dictates.</p>
<blockquote>
<p>Recall from previous posts that <code>|params| body</code> is the syntax for Rust closures. <code>bench.iter()</code> takes a closure with zero parameters.</p>
</blockquote>
<p>Now we can run the benchmark with <code>cargo bench</code>:</p>
<pre class="highlight shell_session"><code><span class="w">$ </span><span class="nc">cargo</span><span class="kv"> bench
</span> Compiling wordutils v0.1.0 (file:///Users/mbutcher/Code/Rust/wordutils)
Finished release [optimized] target(s) in 2.33s
Running target/release/deps/wordutils-0770f6e0ea5ebd16
running 1 test
test tests::bench_initials ... bench: 512 ns/iter (+/- 53)
test result: ok. 0 passed; 0 failed; 0 ignored; 1 measured; 0 filtered out
</code></pre>
<p>What we see from the result is that our <code>initials</code> test took about 512ns per operation, with a variance of 53. (I'm actually a little surprised. This is 4x faster than my Go implementation.)</p>
<p>As benchmarking is still fairly new, to learn more you'll need to take a peek at the <a href="https://doc.rust-lang.org/1.4.0/book/benchmark-tests.html">nightly documentation on this feature</a>.</p>
<h2>Documentation Testing</h2>
<p>While benchmarking is still a new feature of Rust, writing and testing examples is a stable feature.</p>
<p>I'm not going to beat around the bush about this, but I think Rust's documentation testing feature is a thing of beauty. Why? Because examples are embedded in the documentation blocks, and executed automatically. When I'm in the process of writing code, I feel like this is more amenable to my coding practice.</p>
<p>My biggest complaint with Go examples is that because they require a context switch, a special method signature, and special markup at the end, they are weird to write. Most Go developers simply don't write examples this way. Also, because Go's example methodology is a biased toward string testing, it's hard to make these examples reflective of real usage.</p>
<p>Rust goes the other direction: Examples are written as part of source code comments, and are executed almost like mini-programs.</p>
<p>But to understand how this testing works, we need to spend a moment on Rust source code documentation.</p>
<h3>Documenting Rust</h3>
<p>Like Go (and many other languages), Rust supports extracting documentation from the source code. Rust uses <a href="https://doc.rust-lang.org/rust-by-example/meta/doc.html">Markdown as the documentation format</a>, which means we can write docs that are a little richer than Go's when it comes to formatting.</p>
<p>Let's document our <code>initials()</code> function:</p>
<pre class="highlight rust"><code><span class="c">/// Given a string, extract the initials.</span>
<span class="c">/// </span>
<span class="c">/// Initials are composed of the first letter of each word, capitalized.</span>
<span class="c">/// They are then joined together with no spaces.</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="nf">initials</span><span class="p">(</span><span class="n">phrase</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">)</span> <span class="k">-></span> <span class="nb">String</span> <span class="p">{</span>
<span class="n">phrase</span><span class="nf">.split_whitespace</span><span class="p">()</span><span class="nf">.map</span><span class="p">(</span>
<span class="p">|</span><span class="n">word</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">|</span> <span class="n">word</span><span class="nf">.chars</span><span class="p">()</span><span class="nf">.next</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">()</span>
<span class="p">)</span><span class="py">.collect</span><span class="p">::</span><span class="o"><</span><span class="nb">String</span><span class="o">></span><span class="p">()</span><span class="nf">.to_uppercase</span><span class="p">()</span>
<span class="p">}</span>
</code></pre>
<p>Comments use three slashes (<code>///</code>). The first line is a complete, punctuated sentence. Unlike Go documentation, this does <em>not</em> begin with the name of the item being documented.</p>
<p>After a line break, we can add a more complete description. Then, to generate the documentation, we run <code>cargo docs</code>. The resulting documentation will be written into the <code>target/doc/wordutils</code> directory as HTML. We'll see an example shortly.</p>
<h3>Adding an example in Markdown</h3>
<p>The idea of Rust's example documentation is that it should accurately replicate how the function would be called in context. So we just embed a snippet of code right into the Markdown, using the sort of conventions you would normally use:</p>
<pre class="highlight rust"><code><span class="c">/// Given a string, extract the initials.</span>
<span class="c">/// </span>
<span class="c">/// Initials are composed of the first letter of each word, capitalized.</span>
<span class="c">/// They are then joined together with no spaces.</span>
<span class="c">/// </span>
<span class="c">/// # Example</span>
<span class="c">/// </span>
<span class="c">/// ```rust</span>
<span class="c">/// let out = wordutils::initials("hello beautiful world");</span>
<span class="c">/// assert_eq!(out, "HBW");</span>
<span class="c">/// ```</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="nf">initials</span><span class="p">(</span><span class="n">phrase</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">)</span> <span class="k">-></span> <span class="nb">String</span> <span class="p">{</span>
<span class="n">phrase</span><span class="nf">.split_whitespace</span><span class="p">()</span><span class="nf">.map</span><span class="p">(</span>
<span class="p">|</span><span class="n">word</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">|</span> <span class="n">word</span><span class="nf">.chars</span><span class="p">()</span><span class="nf">.next</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">()</span>
<span class="p">)</span><span class="py">.collect</span><span class="p">::</span><span class="o"><</span><span class="nb">String</span><span class="o">></span><span class="p">()</span><span class="nf">.to_uppercase</span><span class="p">()</span>
<span class="p">}</span>
</code></pre>
<p>Unlike our unit tests, we do need to call the package by its full name (or use a <code>use</code>).</p>
<p>Rust will run the inlined examples as unit tests during the testing phase:</p>
<pre class="highlight shell_session"><code><span class="w">$ </span><span class="nc">cargo</span><span class="kv"> test
</span> Compiling wordutils v0.1.0 (file:///Users/mbutcher/Code/Rust/wordutils)
Finished dev [unoptimized + debuginfo] target(s) in 1.07s
Running target/debug/deps/wordutils-e01fe756921f1114
running 1 test
test tests::bench_initials ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Doc-tests wordutils
running 1 test
test src/lib.rs - initials (line 11) ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
</code></pre>
<p>In the <code>Doc-tests</code> section, we can see that it ran our documentation test. (Interestingly, it also appears to have run <code>bench_intials</code> to make sure that it didn't fail.)</p>
<p>A documentation test is considered successful if it doesn't panic. (It does not, however, check whether one <a href="https://en.wikipedia.org/wiki/Phrases_from_The_Hitchhiker%27s_Guide_to_the_Galaxy#Don't_Panic">knows where one's towel is</a>.)</p>
<p>You can also show examples of failures by annotating the markdown code block with <code>should_panic</code>. This test is totally contrived, but shows the basic idea:</p>
<pre class="highlight rust"><code><span class="c">/// Given a string, extract the initials.</span>
<span class="c">/// </span>
<span class="c">/// Initials are composed of the first letter of each word, capitalized.</span>
<span class="c">/// They are then joined together with no spaces.</span>
<span class="c">/// </span>
<span class="c">/// # Example</span>
<span class="c">/// </span>
<span class="c">/// ```rust</span>
<span class="c">/// let out = wordutils::initials("hello beautiful world");</span>
<span class="c">/// assert_eq!(out, "HBW");</span>
<span class="c">/// ```</span>
<span class="c">/// </span>
<span class="c">/// # Panics</span>
<span class="c">/// </span>
<span class="c">/// ```rust,should_panic</span>
<span class="c">/// let out = wordutils::initials("");</span>
<span class="c">/// assert_eq!(out, "hello");</span>
<span class="c">/// ```</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="nf">initials</span><span class="p">(</span><span class="n">phrase</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">)</span> <span class="k">-></span> <span class="nb">String</span> <span class="p">{</span>
<span class="n">phrase</span><span class="nf">.split_whitespace</span><span class="p">()</span><span class="nf">.map</span><span class="p">(</span>
<span class="p">|</span><span class="n">word</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">|</span> <span class="n">word</span><span class="nf">.chars</span><span class="p">()</span><span class="nf">.next</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">()</span>
<span class="p">)</span><span class="py">.collect</span><span class="p">::</span><span class="o"><</span><span class="nb">String</span><span class="o">></span><span class="p">()</span><span class="nf">.to_uppercase</span><span class="p">()</span>
<span class="p">}</span>
</code></pre>
<p>In the second example, the <code>assert_eq!</code> will panic because the initials of <code>""</code> will not be <code>"hello"</code>. But when we run the test, it will succeed because it was expecting that example to panic.</p>
<pre class="highlight shell_session"><code><span class="w">$ </span><span class="nc">cargo</span><span class="kv"> test
</span> Compiling wordutils v0.1.0 (file:///Users/mbutcher/Code/Rust/wordutils)
Finished dev [unoptimized + debuginfo] target(s) in 2.55s
Running target/debug/deps/wordutils-e01fe756921f1114
running 1 test
test tests::bench_initials ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Doc-tests wordutils
running 2 tests
test src/lib.rs - initials (line 11) ... ok
test src/lib.rs - initials (line 18) ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
</code></pre>
<p>Now if we generate the documentation with <code>cargo doc</code>, this is what it will look like:</p>
<p><img src="/img/rust-example-docs.png" alt="Cargo Docs" width="800" height="425" /></p>
<p>At this point, we've seen both benchmarking and documentation testing. And as with Go, we haven't explored every nook and cranny of the framework, but we've gotten the basic idea.</p>
<p>However, we have one more thing to cover. And this is a feature that Go doesn't support in its core toolset: Integration testing.</p>
<h2>Integration Tests in Rust</h2>
<p>The last category of tests to cover is integration tests. Typically, while unit tests cover individual functions and are "close to the source", integration tests are designed to show that, from an outside perspective, things work as advertised.</p>
<p>Often, unit tests will use fixtures and mocks to test just very specific parts of the code. Integration testing more often forgoes mocks (at least mocks of internal things), and tests that the code is functioning together as a whole.</p>
<p>Rust has a top-level concept of integration tests. Inside of your Cargo project, they are placed in the <code>tests/</code> directory adjacent to <code>src/</code> and <code>target/</code>:</p>
<pre class="highlight plaintext"><code>.
├── Cargo.lock
├── Cargo.toml
├── src
│  ├── lib.rs
│  └── tests.rs
├── target
│  ├── debug
│  ├── doc
│  └── release
└── tests
└── integration_tests.rs
</code></pre>
<p>Integration tests are constructed the way the unit tests and documentation tests are: Write some code, use some asserts to make sure it does what it is supposed to.</p>
<p>Unfortunately for us, we don't have a whole lot of "integration" to do. But we'll still see the main pattern for integration tests, and see how this differs from unit tests.</p>
<p>I created an integration test in <code>tests/</code> named <code>integration_tests.rs</code>. Zero points for originality.</p>
<pre class="highlight rust"><code><span class="k">extern</span> <span class="n">crate</span> <span class="n">wordutils</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">wordutils</span><span class="p">::</span><span class="n">initials</span><span class="p">;</span>
<span class="cp">#[test]</span>
<span class="k">fn</span> <span class="nf">do_initials</span><span class="p">()</span> <span class="p">{</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="nf">initials</span><span class="p">(</span><span class="s">"j. alfred prufrock"</span><span class="p">),</span> <span class="s">"JAP"</span><span class="p">);</span>
<span class="p">}</span>
</code></pre>
<p>The main thing to notice about this is that integration tests are structured the same way that an external tool would use the library.</p>
<blockquote>
<p><strong>Pro Tip:</strong> That means that the contents of a crate's <code>tests/</code> directory is a great place to figure out how to use a library.</p>
</blockquote>
<p>So we use <code>extern</code> to declare that we are using <code>wordutils</code> and we use <code>use</code> to import the <code>initials</code> function into our current namespace.</p>
<p>However, we still have to use the <code>#[test]</code> attribute to declare that <code>do_initials()</code> is a test function.</p>
<p>As with unit and documentation tests, Cargo will do us the favor of running these tests as part of <code>cargo test</code>:</p>
<pre class="highlight shell_session"><code><span class="w">$ </span><span class="nc">cargo</span><span class="kv"> test
</span> Compiling wordutils v0.1.0 (file:///Users/mbutcher/Code/Rust/wordutils)
Finished dev [unoptimized + debuginfo] target(s) in 0.72s
Running target/debug/deps/wordutils-e01fe756921f1114
running 1 test
test tests::bench_initials ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Running target/debug/deps/integration_tests-543eeef678725b84
running 1 test
test do_initials ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Doc-tests wordutils
running 2 tests
test src/lib.rs - initials (line 11) ... ok
test src/lib.rs - initials (line 18) ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
</code></pre>
<p>To be completely honest, as a Rust neophyte, I am not entirely sure how beneficial it will be to have integration test support built-in like this. But I do like the idea that I can look in a crate's <code>tests/</code> directory and get an idea of how the public API is <em>supposed to work</em>.</p>
<h2>Conclusion</h2>
<p>In this fifth post of the series, we have looked at three additional types of testing in Rust:</p>
<ul>
<li>Benchmarking</li>
<li>Examples in documentation</li>
<li>Integration tests</li>
</ul>
<p>For the most part, Rust's testing strategy is not much different than Go's. We don't see drastically different paradigms or conventions. But what we do see is perhaps a greater "ergonomic" in Rust, where examples are <em>inside of the documentation</em> instead of in the unit tests, and where integration tests are separated from unit tests, and designed to mirror the user experience of developers who use your libraries.</p>
Using Azure Static Websiteshttp://technosophos.com/2018/07/21/using-azure-static-websites.html2018-07-21T20:06:00+00:002018-07-25T14:35:08+00:00Article Author<p>Over the years I've chronicled the technical changes to this blog and its hosting provider. Years ago, I moved it <a href="http://technosophos.com/2013/11/17/migrating-from-drupal-to-middleman.html">from Drupal to Middleman</a> to cut down on the maintenance. Later I <a href="http://technosophos.com/2017/03/18/dockerizing-ruby-to-stay-sane.html">containerized my Ruby environment</a> to get rid of RVM madness</p>
<p>For a fun weekend project, I moved from S3 to the brand new (still in preview) <a href="https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website">Static Websites</a> service from Azure.</p>
<p>Essentially, I moved the hosted portion of my blog, but didn't actually change the underlying blog software. I'm still using Middleman to generate the blog.</p>
<p>In this post, I'll cover the first two steps of my move:</p>
<ul>
<li>Move my blog's files from S3 to Azure Storage</li>
<li>Use Azure Static Websites to serve the site</li>
</ul>
<p>In the future, I want to add a couple new features, though:</p>
<ul>
<li>Add Azure CDN to speed things up</li>
<li>Finally start using SSL for this site</li>
</ul>
<p>So I will cover those in a follow-up post.</p>
<h2>Step 1: Setting Up Azure Storage</h2>
<p>Really, the process of "migration" is more about uploading a bunch of static files, then redirecting DNS. Because <a href="https://middlemanapp.com/">Middleman</a> can generate the entire site from my source code (which lives in a private BitBucket repo), there's no data migration necessary. I don't have to get data out of S3. I can just upload a fresh copy.</p>
<p>So to kick things off, I logged into the <a href="https://portal.azure.com">Azure web portal</a> and then created a new storage account. And I turned on the <em>Static Website</em> feature.</p>
<p>Rather than repeat the exact steps I did, I'll point you straight to the (frequently updated) <a href="https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website">official documentation</a>. That team does a stellar job of keeping up with changes, and for a preview service that is important. The entire process documented there seriously took me only a couple of minutes.</p>
<p>A few quick notes:</p>
<ul>
<li>When the docs tell you to enter <code>index.html</code>, which seems to override the default of <code>index.html</code>, you do actually need to do that. I suspect that will be fixed in the future.</li>
<li>I did not create a <code>$web</code> container at this point. I let the tooling do it for me later.</li>
</ul>
<h2>Step 2: Upload My Site</h2>
<p>I have a handy <code>Makefile</code> that I use to do various blog tasks. It looks like this:</p>
<pre class="highlight plaintext"><code>APPROOT=/usr/src/myapp
UPLOAD_FLAGS ?= -o table
.PHONY: post
post: TITLE ?= Untitled
post: COMMAND = bundle exec middleman article "$(TITLE)"
post: dockerize
.PHONY: build
build: COMMAND = bundle exec middleman build
build: dockerize
.PHONY: serve
serve: EFLAGS = -p 4567:4567
serve: COMMAND = bundle exec middleman serve
serve: dockerize
.PHONY: docker-build
docker-build:
docker build -t $(IMAGE) .
.PHONY: dockerize
dockerize:
docker run -it --rm --name $(NAME) -v "$(CURDIR)":$(APPROOT) -w $(APPROOT) $(EFLAGS) $(IMAGE) $(COMMAND)
.PHONY: dist
dist:
# Code to send this to AWS
</code></pre>
<p>In a nutshell:</p>
<ul>
<li><code>post</code> creates a new post</li>
<li><code>serve</code> starts a local testing server</li>
<li><code>build</code> generates a static version of the site</li>
<li><code>dist</code> sends the static site to S3</li>
</ul>
<p>To this <code>Makefile</code> I just added a new target:</p>
<pre class="highlight plaintext"><code>.PHONY: upload
upload:
az storage blob upload-batch -d '$$web' -s build/ $(UPLOAD_FLAGS)
</code></pre>
<p>(<code>UPLOAD_FLAGS</code> just gives me a way to override the flags from the command line, like <code>$ UPLOAD_FLAGS="--dry-run" make upload</code>.)</p>
<p>The new command does a bulk upload of my static site (<code>az storage blob upload-batch</code>) sending it to the destination (<code>-d</code>) container named <code>$web</code> (note that we escaped this for Make by doing <code>$$</code>). And it reads the sources from the <code>build/</code> folder.</p>
<p>To authenticate <code>az</code> to my account, I set the env var <code>AZURE_STORAGE_CONNECTION_STRING</code>.</p>
<p>Now running <code>make build upload</code> builds my site from source, then uploads it to my new Azure static website.</p>
<p>The first time I ran the upload, it created the <code>$web</code> container, but it seemed to take about three minutes to get everything synced. In particular mapping the <code>index.html</code> file to the document root seemed to take a bit. But from there, everything worked as expected.</p>
<h2>Where Next?</h2>
<p>At this point, I can hit my Technosophos blog at the URL provided by Azure. There are two possible routes to go from here:</p>
<ul>
<li>I could set up Azure's DNS service to point directly to this endpoint. This process is actually pretty easy. But that's not what I want to do.</li>
<li>I would like to set up Azure's CDN service to cache my blog, then add an SSL certificate on the CDN service (something not supported by static websites yet) so that the blog will be fully TLS.</li>
</ul>
<p>That second option is what I am exploring now, and will document in a future post.</p>
From Go to Rust - Unit Testinghttp://technosophos.com/2018/07/07/from-go-to-rust-testing.html2018-07-07T22:46:00+00:002018-07-09T03:16:27+00:00Article Author<p>In this fourth installment of the series, we'll transform some Go tests into Rust tests.</p>
<p>In case you missed anything:</p>
<ul>
<li>In the <a href="/2018/05/27/the-go-developers-quickstart-guide-to-rust.html">first part</a> of this series, we looked at some fundamentals of Rust, and how they compare to Go.</li>
<li>In the <a href="/2018/06/04/from-go-to-rust-with-an-http-server.html">second part</a> we took a Go web server and reimplemented it in Rust.</li>
<li>The <a href="/2018/06/12/from-go-to-rust-json-and-yaml.html">third part</a> focused on JSON and serialization, as we compared Go's annotation-based encoding to Rust's Serde libraries.</li>
</ul>
<p>Now in this part, we'll look at testing. Along the way, we'll also see:</p>
<ul>
<li>How to create a library package in Rust</li>
<li>How to use modules, and how they compare to Go packages</li>
<li>How to use Rust's assertion tools</li>
<li>How to run tests from Cargo</li>
</ul>
<h2>Go Did Testing Right... Mostly</h2>
<p>I am a big fan of Go's approach to testing. Tests are easy to write, easy to run, and live alongside the stuff that they test. Adding benchmarking support was cool. And I like the way that documentation functions get automatically tested (though the implementation is limited to nearly trivial functions).</p>
<p>I also like the fact that Go makes it easy to test private (unexported) functions. I know there's some dogma involved here. I've heard people adamantly claim that private functions should not be tested. But for purely pragmatic reasons, my view is that we <em>should</em> be able to test whatever we want.</p>
<p>But there are a few things about Go's built-in testing that I'm not terribly keen on.</p>
<p>I'll never understand why the Go developers didn't just add an assertions library to the testing library. I've heard the "asserts get abused" line, but I don't find it convincing. That said, it's a shortcoming easily remedied by a <a href="https://github.com/stretchr/testify">decent assert library</a>.</p>
<p>One thing I'm not a fan of in general, though, is using "magic" prefixes or suffixes to determine how to execute a function. I think function name scanning sets a dangerous precedent for how reflection ought to be used. And I find that in practice it results in an arbitrary limitation on what I can actually name my functions. But in spite of my general disdain for that pattern, I've been happy with the way it works in Go's testing suite.</p>
<p>Overall, I think Go makes it amazingly simple to write tests. And compared to languages like Java, Python, JavaScript, and PHP, working with Go tests is a breeze.</p>
<p>So when diving into Rust (reminder: This really is my first go-around with the language), I've been interested to see how Rust's testing stacks up. In this article, we'll focus on unit tests.</p>
<h2>Something To Test: Wordutils</h2>
<p>For the past few posts, I've been writing small programs. But with testing as the focus, it seems like the right time to try my hand at writing a library. So here's a small <code>wordutils</code> library:</p>
<pre class="highlight go"><code><span class="k">package</span><span class="x"> </span><span class="n">wordutils</span><span class="x">
</span><span class="k">import</span><span class="x"> </span><span class="p">(</span><span class="x">
</span><span class="s">"bufio"</span><span class="x">
</span><span class="s">"strings"</span><span class="x">
</span><span class="p">)</span><span class="x">
</span><span class="c">// Initials returns a string with the first letter of each word in the given string.</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">Initials</span><span class="p">(</span><span class="n">phrase</span><span class="x"> </span><span class="kt">string</span><span class="p">)</span><span class="x"> </span><span class="p">(</span><span class="kt">string</span><span class="p">,</span><span class="x"> </span><span class="kt">error</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">wrds</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">words</span><span class="p">(</span><span class="n">phrase</span><span class="p">)</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="s">""</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">initials</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="s">""</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">_</span><span class="p">,</span><span class="x"> </span><span class="n">word</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="k">range</span><span class="x"> </span><span class="n">wrds</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">initials</span><span class="x"> </span><span class="o">+=</span><span class="x"> </span><span class="n">word</span><span class="p">[</span><span class="m">0</span><span class="o">:</span><span class="m">1</span><span class="p">]</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="n">strings</span><span class="o">.</span><span class="n">ToUpper</span><span class="p">(</span><span class="n">initials</span><span class="p">),</span><span class="x"> </span><span class="no">nil</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">words</span><span class="p">(</span><span class="n">str</span><span class="x"> </span><span class="kt">string</span><span class="p">)</span><span class="x"> </span><span class="p">([]</span><span class="kt">string</span><span class="p">,</span><span class="x"> </span><span class="kt">error</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">wordList</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">{}</span><span class="x">
</span><span class="n">scanner</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">bufio</span><span class="o">.</span><span class="n">NewScanner</span><span class="p">(</span><span class="n">strings</span><span class="o">.</span><span class="n">NewReader</span><span class="p">(</span><span class="n">str</span><span class="p">))</span><span class="x">
</span><span class="n">scanner</span><span class="o">.</span><span class="n">Split</span><span class="p">(</span><span class="n">bufio</span><span class="o">.</span><span class="n">ScanWords</span><span class="p">)</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">scanner</span><span class="o">.</span><span class="n">Scan</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">wordList</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="nb">append</span><span class="p">(</span><span class="n">wordList</span><span class="p">,</span><span class="x"> </span><span class="n">scanner</span><span class="o">.</span><span class="n">Text</span><span class="p">())</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="n">wordList</span><span class="p">,</span><span class="x"> </span><span class="n">scanner</span><span class="o">.</span><span class="n">Err</span><span class="p">()</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre>
<p>The library above declares two functions: <code>words</code>, which splits a string into words, and <code>Initials</code>, which returns a string that is the capitalized version of the first letter of each word in the given string. </p>
<p>While there's no Go standard library function that does what <code>words()</code> does, I decided to write it as an internal function (non-exported) to give me a point of comparison with Rust. When re-implementing, we'll follow the same scoping rules.</p>
<p>These two functions are easy enough to test. So alongside <code>wordutils.go</code>, here is the contents of <code>wordutils_test.go</code>:</p>
<pre class="highlight go"><code><span class="k">package</span><span class="x"> </span><span class="n">wordutils</span><span class="x">
</span><span class="k">import</span><span class="x"> </span><span class="p">(</span><span class="x">
</span><span class="s">"testing"</span><span class="x">
</span><span class="p">)</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">TestWords</span><span class="p">(</span><span class="n">t</span><span class="x"> </span><span class="o">*</span><span class="n">testing</span><span class="o">.</span><span class="n">T</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">in</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="s">"this is the way</span><span class="se">\n</span><span class="s"> the world ends"</span><span class="x">
</span><span class="n">out</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">words</span><span class="p">(</span><span class="n">in</span><span class="p">)</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">t</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">expect</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">"this"</span><span class="p">,</span><span class="x"> </span><span class="s">"is"</span><span class="p">,</span><span class="x"> </span><span class="s">"the"</span><span class="p">,</span><span class="x"> </span><span class="s">"way"</span><span class="p">,</span><span class="x"> </span><span class="s">"the"</span><span class="p">,</span><span class="x"> </span><span class="s">"world"</span><span class="p">,</span><span class="x"> </span><span class="s">"ends"</span><span class="p">}</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="nb">len</span><span class="p">(</span><span class="n">out</span><span class="p">)</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="nb">len</span><span class="p">(</span><span class="n">expect</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">t</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="s">"expected same length"</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">i</span><span class="p">,</span><span class="x"> </span><span class="n">word</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="k">range</span><span class="x"> </span><span class="n">out</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">word</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="n">expect</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">t</span><span class="o">.</span><span class="n">Errorf</span><span class="p">(</span><span class="s">"expected word %d to be %q, got %q"</span><span class="p">,</span><span class="x"> </span><span class="n">i</span><span class="p">,</span><span class="x"> </span><span class="n">expect</span><span class="p">[</span><span class="n">i</span><span class="p">],</span><span class="x"> </span><span class="n">word</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">TestInitials</span><span class="p">(</span><span class="n">t</span><span class="x"> </span><span class="o">*</span><span class="n">testing</span><span class="o">.</span><span class="n">T</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">in</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="s">"not with a bang but a whimper"</span><span class="x">
</span><span class="n">expect</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="s">"NWABBAW"</span><span class="x">
</span><span class="n">out</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">Initials</span><span class="p">(</span><span class="n">in</span><span class="p">)</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">t</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">out</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="n">expect</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">t</span><span class="o">.</span><span class="n">Errorf</span><span class="p">(</span><span class="s">"expected %q, got %q"</span><span class="p">,</span><span class="x"> </span><span class="n">expect</span><span class="p">,</span><span class="x"> </span><span class="n">out</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre>
<p>There's a simple unit test for each of the functions we wrote. These tests aren't terribly robust (they don't test the error cases), but they're good enough for us to start modeling a Rust implementation.</p>
<h2>Creating a Rust Library</h2>
<p>Instead of creating an executable with <code>cargo new --bin</code>, we're going to create a new library. And as a bonus... I just learned that we can initialize a Git repo as part of package creation:</p>
<pre class="highlight shell_session"><code><span class="w">$ </span><span class="nc">cargo</span><span class="kv"> new --vcs git --lib wordutils
</span> Created library `wordutils` project
</code></pre>
<p>The <code>--lib</code> flag sets up the package as a library:</p>
<pre class="highlight plaintext"><code>wordutils
├── Cargo.lock
├── Cargo.toml
├── src
│  └── lib.rs
└── target
└── debug
└── ...
</code></pre>
<p>In previous articles, we worked on <code>src/main.rs</code>. Note that with <code>--lib</code>, the file created for us is <code>src/lib.rs</code>. If we take a look inside of it, we'll see that some code was already created for us:</p>
<pre class="highlight rust"><code><span class="err">#</span><span class="p">[</span><span class="nf">cfg</span><span class="p">(</span><span class="n">test</span><span class="p">)]</span>
<span class="k">mod</span> <span class="n">tests</span> <span class="p">{</span>
<span class="cp">#[test]</span>
<span class="k">fn</span> <span class="nf">it_works</span><span class="p">()</span> <span class="p">{</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="mi">2</span> <span class="o">+</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
<p>Huh... it's a test scaffold. It's like <em>they knew what I was planning</em>. Let's run the tests just to see what happens. From within the <code>wordutils</code> directory, we just run <code>cargo test</code>:</p>
<pre class="highlight shell_session"><code>cargo test
Compiling wordutils v0.1.0 (file:///Users/mbutcher/Code/Rust/wordutils)
Finished dev [unoptimized + debuginfo] target(s) in 4.24 secs
Running target/debug/deps/wordutils-9a757f9d84faff12
running 1 test
test tests::it_works ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Doc-tests wordutils
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
</code></pre>
<p>Alright, lest I get ahead of myself... we know that the initial test works. Let's get about the business of writing the library, and we'll return to testing later.</p>
<h2>Wordutils: The Rust Version</h2>
<p>Again, this is my first attempt at writing a Rust library. So my initial reaction to the contents of <code>lib.rs</code> was, "Wait... if the tests are in <em>there</em>, where do I put my code?" It turns out that for simple modules like ours, the answer is: put the code and the tests in the same file.</p>
<p>We'll do that first, then later look at ways of breaking things up. To start, we'll leaving the stub test alone and adding our new functions.</p>
<pre class="highlight rust"><code><span class="k">fn</span> <span class="nf">words</span><span class="p">(</span><span class="n">phrase</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">)</span> <span class="k">-></span> <span class="nn">std</span><span class="p">::</span><span class="nn">str</span><span class="p">::</span><span class="n">SplitWhitespace</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">phrase</span><span class="nf">.split_whitespace</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="nf">initials</span><span class="p">(</span><span class="n">phrase</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">)</span> <span class="k">-></span> <span class="nb">String</span> <span class="p">{</span>
<span class="nf">words</span><span class="p">(</span><span class="n">phrase</span><span class="p">)</span><span class="nf">.map</span><span class="p">(</span>
<span class="p">|</span><span class="n">word</span><span class="p">|</span> <span class="n">word</span><span class="nf">.chars</span><span class="p">()</span><span class="nf">.next</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">()</span>
<span class="p">)</span><span class="py">.collect</span><span class="p">::</span><span class="o"><</span><span class="nb">String</span><span class="o">></span><span class="p">()</span><span class="nf">.to_uppercase</span><span class="p">()</span>
<span class="p">}</span>
<span class="cp">#[cfg(test)]</span>
<span class="k">mod</span> <span class="n">tests</span> <span class="p">{</span>
<span class="cp">#[test]</span>
<span class="k">fn</span> <span class="nf">it_works</span><span class="p">()</span> <span class="p">{</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="mi">2</span> <span class="o">+</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
<p>We've replicated our <code>wordutils.go</code> functionality in just a few lines of code. And it's exciting, because we've encountered a few new concepts that haven't appeared in previous installments of the series.</p>
<h3>The <code>words()</code> function</h3>
<p>The <code>words()</code> function is a oneliner because Rust's standard library already has a word splitter.</p>
<pre class="highlight rust"><code><span class="k">fn</span> <span class="nf">words</span><span class="p">(</span><span class="n">phrase</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">)</span> <span class="k">-></span> <span class="nn">std</span><span class="p">::</span><span class="nn">str</span><span class="p">::</span><span class="n">SplitWhitespace</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">phrase</span><span class="nf">.split_whitespace</span><span class="p">()</span>
<span class="p">}</span>
</code></pre>
<p>There are two things to note about this function. </p>
<p>First, we return a type called <code>std::str::SplitWhitespace</code>. The <code>SplitWhitespace</code> type is the type returned by <code>split_whitespace</code>. It has several traits associated with it that make it useful as a type.</p>
<p>In Go, it is common to keep the number of types sparse. For something like <code>split_whitespace</code>, Go developers would likely return a <code>[]string</code> (slice of strings). But in Rust, it is common to return specialized types that can then implement multiple traits. In doing this (as we will see in a moment), we gain added flexibility that leads to concise and readable code.</p>
<p>Second, I chose to return <code>std::str::SplitWhitespace</code> instead of adding a <code>use std::str::SplitWhitespace</code> at the top, and then just returning <code>SplitWhitespace</code>. Either way works, but it is probably more common to add the <code>use</code> line rather than use a fully qualified name in a return value.</p>
<h3>The <code>initials()</code> function</h3>
<p>The second function declared is <code>initials()</code>. Normally, we'd actually write this as a one-liner, too. But since we're all new to Rust, I figured it would be more readable to expand it into a three-line body:</p>
<pre class="highlight rust"><code><span class="k">pub</span> <span class="k">fn</span> <span class="nf">initials</span><span class="p">(</span><span class="n">phrase</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">)</span> <span class="k">-></span> <span class="nb">String</span> <span class="p">{</span>
<span class="nf">words</span><span class="p">(</span><span class="n">phrase</span><span class="p">)</span><span class="nf">.map</span><span class="p">(</span>
<span class="p">|</span><span class="n">word</span><span class="p">|</span> <span class="n">word</span><span class="nf">.chars</span><span class="p">()</span><span class="nf">.next</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">()</span>
<span class="p">)</span><span class="py">.collect</span><span class="p">::</span><span class="o"><</span><span class="nb">String</span><span class="o">></span><span class="p">()</span><span class="nf">.to_uppercase</span><span class="p">()</span>
<span class="p">}</span>
</code></pre>
<p>Notice that this function definition begins with <code>pub</code>. That tells the Rust toolchain that this function is a <em>public</em> (exported) function. Unlike Go, capitalization makes no difference as to the visibility of a function (or anything else). In Rust, modules must use <code>pub</code> to mark a method as visible outside of the module.</p>
<p>Like Go, though, Rust only has two visibilities: public (with <code>pub</code>) and private (the default). But the rules for privacy are slightly different in Rust than in Go. According to <a href="https://doc.rust-lang.org/book/second-edition/ch07-02-controlling-visibility-with-pub.html#privacy-rules">the Rust visibility documentation</a>:</p>
<blockquote>
<p>If an item is private, it can be accessed only by its immediate parent module and any of the parent’s child modules.</p>
</blockquote>
<p>In contrast, privacy in Go dictates that <em>only the present package</em> may access a private item. We'll see in a moment why this nuance makes a difference when we write Rust tests.</p>
<p>Now let's look more closely at the function chain we run inside of <code>initials()</code>:</p>
<pre class="highlight rust"><code><span class="nf">words</span><span class="p">(</span><span class="n">phrase</span><span class="p">)</span><span class="nf">.map</span><span class="p">(</span>
<span class="p">|</span><span class="n">word</span><span class="p">|</span> <span class="n">word</span><span class="nf">.chars</span><span class="p">()</span><span class="nf">.next</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">()</span>
<span class="p">)</span><span class="py">.collect</span><span class="p">::</span><span class="o"><</span><span class="nb">String</span><span class="o">></span><span class="p">()</span><span class="nf">.to_uppercase</span><span class="p">()</span>
</code></pre>
<p>We start by running the <code>words()</code> function we saw above. The <code>SplitWhitespace</code> object returned from that implements <a href="https://doc.rust-lang.org/std/iter/trait.Iterator.html">Iterator</a>. Because of this, we could toss the result into a <code>for</code> loop:</p>
<pre class="highlight rust"><code><span class="k">for</span> <span class="n">word</span> <span class="n">in</span> <span class="nf">words</span><span class="p">(</span><span class="s">"Mary had a little lamb"</span><span class="p">)</span> <span class="p">{</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"Word: {}"</span><span class="p">,</span> <span class="n">word</span><span class="p">);</span>
<span class="p">}</span>
</code></pre>
<p>See what we did? The <code>SplitWhitespace</code> type is an iterator (implements <code>std::iter::Iterator</code>), so we can use it in a <code>for</code> loop <a href="https://doc.rust-lang.org/rust-by-example/flow_control/for.html#for-and-range">without doing anything special</a>. Go does not have a concept of an iterator, so this code might look surprising.</p>
<p>Rust iterators are more than just a convenience for <code>for</code> loops, though. An <code>Iterator</code> has around a dozen useful functions attached to it. And one of them is <code>map()</code>. The <code>map()</code> function takes a closure (inline function), runs it on every item in the iterator, and returns the results as a new iterator.</p>
<p>In Rust, closures look different than regular functions. They take the form:</p>
<pre class="highlight rust"><code><span class="p">|</span><span class="n">param1</span><span class="p">,</span> <span class="n">param2</span><span class="p">,</span> <span class="err">...</span><span class="p">|</span> <span class="n">function_body</span>
</code></pre>
<p>We can spread the function body out into a block, if we'd like:</p>
<pre class="highlight rust"><code><span class="p">|</span><span class="n">param1</span><span class="p">|</span> <span class="p">{</span>
<span class="n">stuff</span><span class="p">;</span>
<span class="n">more_stuff</span><span class="p">;</span>
<span class="n">return_val</span>
<span class="p">}</span>
</code></pre>
<p>In our code, we call the <code>map()</code> function and give it a transformation: It takes a word, and returns the first character of that word.</p>
<pre class="highlight rust"><code><span class="p">|</span><span class="n">word</span><span class="p">|</span> <span class="n">word</span><span class="nf">.chars</span><span class="p">()</span><span class="nf">.next</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">()</span>
</code></pre>
<p>Note that here (as in many cases in Rust), we let the compiler infer the type of <code>word</code>. The function is still type safe because Rust can determine at compile time that anything assigned to <code>word</code> will be a string. (If the type were ambiguous for some reason, we could annotate it <code>|word: &str| ...</code>)</p>
<p>So <code>word.chars().next()</code> essentially says "convert this word string to a list of characters, then use <code>next()</code> to pop the first character." But <code>next()</code> is safe: Instead of returning a character, it returns an <code>Option<char></code>. If there is no character to return, it will send back a <code>None</code>. We happen to know that all of the words returned from <code>split_whitespace()</code> have at least one character in them. So instead of testing whether the result of <code>next()</code> was a <code>Some<char></code> or a <code>None</code>, we can use <code>unwrap()</code> to just get the <code>char</code> value.</p>
<p>Note that if for some reason we did get a <code>None</code>, <code>unwrap()</code> would cause a panic.</p>
<p>At this point, the <code>words().map()</code> combo has returned an iterator of chars. We want to turn that into a <code>String</code>. Believe it or not, this is really easy in Rust because an <code>Iterator</code> has a function called <code>collect<T>()</code> that takes an iterator and transforms it into some other collection type (<code>T</code>).</p>
<p>We could, for example, use <code>collect::<Vec<char>></code> to collect our iterator into a vector (list) of characters. And in Rust, a <code>String</code> happens to be... wait for it... <em>a collection of characters</em>! So all we need to do to transform our character iterator into a <code>String</code> is call <code>collect::<String>()</code>. (Recall that <code>::<T></code>, the <em>turbofish</em>, tells a function that takes a generic how to fill out that generic.)</p>
<p>That's it for our two functions. Essentially, we've now reimplemented our Go <code>wordutils</code> library. It's time to do some testing.</p>
<h2>Writing the Tests</h2>
<p>We already got a hint of how to write tests when we took our initial look at <code>lib.rs</code>. We saw a basic test that looked like this:</p>
<pre class="highlight rust"><code><span class="err">#</span><span class="p">[</span><span class="nf">cfg</span><span class="p">(</span><span class="n">test</span><span class="p">)]</span>
<span class="k">mod</span> <span class="n">tests</span> <span class="p">{</span>
<span class="cp">#[test]</span>
<span class="k">fn</span> <span class="nf">it_works</span><span class="p">()</span> <span class="p">{</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="mi">2</span> <span class="o">+</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
<p>So let's just roll with it and try to flesh out some actual tests based on that pattern. Here I'm replicating the tests from <code>wordutils_test.go</code> and also adding a few tests for handling empty strings:</p>
<pre class="highlight rust"><code><span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="nn">str</span><span class="p">::</span><span class="n">SplitWhitespace</span><span class="p">;</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="nf">initials</span><span class="p">(</span><span class="n">phrase</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">)</span> <span class="k">-></span> <span class="nb">String</span> <span class="p">{</span>
<span class="nf">words</span><span class="p">(</span><span class="n">phrase</span><span class="p">)</span><span class="nf">.map</span><span class="p">(</span>
<span class="p">|</span><span class="n">word</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">|</span> <span class="n">word</span><span class="nf">.chars</span><span class="p">()</span><span class="nf">.next</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">()</span>
<span class="p">)</span><span class="py">.collect</span><span class="p">::</span><span class="o"><</span><span class="nb">String</span><span class="o">></span><span class="p">()</span><span class="nf">.to_uppercase</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">words</span><span class="p">(</span><span class="n">phrase</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">)</span> <span class="k">-></span> <span class="n">SplitWhitespace</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">phrase</span><span class="nf">.split_whitespace</span><span class="p">()</span>
<span class="p">}</span>
<span class="cp">#[cfg(test)]</span>
<span class="k">mod</span> <span class="n">tests</span> <span class="p">{</span>
<span class="k">use</span> <span class="nn">super</span><span class="p">::</span><span class="o">*</span><span class="p">;</span>
<span class="cp">#[test]</span>
<span class="k">fn</span> <span class="nf">test_words</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">input</span> <span class="o">=</span> <span class="s">"this is the way</span><span class="se">\n</span><span class="s"> the world ends"</span><span class="p">;</span>
<span class="k">let</span> <span class="n">expect</span> <span class="o">=</span> <span class="nd">vec!</span><span class="p">[</span><span class="s">"this"</span><span class="p">,</span> <span class="s">"is"</span><span class="p">,</span> <span class="s">"the"</span><span class="p">,</span> <span class="s">"way"</span><span class="p">,</span> <span class="s">"the"</span><span class="p">,</span> <span class="s">"world"</span><span class="p">,</span> <span class="s">"ends"</span><span class="p">];</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="nf">words</span><span class="p">(</span><span class="n">input</span><span class="p">)</span><span class="py">.collect</span><span class="p">::</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><&</span><span class="nb">str</span><span class="o">>></span><span class="p">(),</span> <span class="n">expect</span><span class="p">)</span>
<span class="p">}</span>
<span class="cp">#[test]</span>
<span class="k">fn</span> <span class="nf">test_initials</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">input</span> <span class="o">=</span> <span class="s">"not with a bang but a whimper"</span><span class="p">;</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="nf">initials</span><span class="p">(</span><span class="n">input</span><span class="p">),</span> <span class="s">"NWABBAW"</span><span class="p">);</span>
<span class="p">}</span>
<span class="cp">#[test]</span>
<span class="k">fn</span> <span class="nf">empty_words</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">input</span> <span class="o">=</span> <span class="s">""</span><span class="p">;</span>
<span class="k">let</span> <span class="n">expect</span><span class="p">:</span> <span class="nb">Vec</span><span class="o"><&</span><span class="nb">str</span><span class="o">></span> <span class="o">=</span> <span class="nn">Vec</span><span class="p">::</span><span class="nf">new</span><span class="p">();</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="nf">words</span><span class="p">(</span><span class="n">input</span><span class="p">)</span><span class="py">.collect</span><span class="p">::</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><&</span><span class="nb">str</span><span class="o">>></span><span class="p">(),</span> <span class="n">expect</span><span class="p">);</span>
<span class="p">}</span>
<span class="cp">#[test]</span>
<span class="k">fn</span> <span class="nf">empty_initials</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">input</span> <span class="o">=</span> <span class="s">""</span><span class="p">;</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="nf">initials</span><span class="p">(</span><span class="n">input</span><span class="p">),</span> <span class="s">""</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
<p>If we run <code>cargo test</code> with the above, we'll see four tests pass. Let's take a quick look at the organization of tests.</p>
<h3>Rust Modules, and the <code>test</code> Module</h3>
<p>In Go, packages are determined largely by the <code>package</code> keyword at the top of each file in a directory. The usual idiom in Go is that a directory name and a package name match. (The history of this is a little complex, as the original version of Go suggested that a directory contains two packages: the base package (<code>foo</code>) and the testing package (<code>foo_test</code>). But that idiom was deprecated shortly after Go 1.0.)</p>
<p>In Rust, when people say "package", they usually mean a crate, which is an entire library or application. Libraries are broken up not into packages, but into <em>modules</em>.</p>
<p>While Go mixes testing and non-testing stuff into the same package, and uses compiler magic to distinguish based on file names, Rust is a little different. In Rust, you store your tests (by convention) in the <code>tests</code> module, decorate the modules with attributes, and then let the toolchain sort and run the tests.</p>
<p>There are noteworthy similarities that Rust and Go share here (especially when contrasted with other popular languages): Tests are stored alongside the code they test. Tests are sorted and executed by the toolchain. Testing support is first-class in the language.</p>
<p>With that in mind, we can take a look at the <code>tests</code> module:</p>
<pre class="highlight rust"><code><span class="err">#</span><span class="p">[</span><span class="nf">cfg</span><span class="p">(</span><span class="n">test</span><span class="p">)]</span>
<span class="k">mod</span> <span class="n">tests</span> <span class="p">{</span>
<span class="k">use</span> <span class="nn">super</span><span class="p">::</span><span class="o">*</span><span class="p">;</span>
<span class="c">// Tests go here</span>
<span class="p">}</span>
</code></pre>
<p>So we've created a module inside of <code>wordutils</code> that is named <code>wordutils::tests</code>. And we've annotated it with <code>#[cfg(test)]</code>, which (if I understand things correctly) is the attribute that tells the compiler to only compile this module during a test run. (Compare this with Go build flags.) Like <code>go test</code>, <code>cargo test</code> compiles a testing binary and then executes it.</p>
<p>See the line <code>use super::*</code>? We've seen <code>use</code> in previous installments of this series. It is used to import names from other modules into the current namespace. Here, we are importing the names from the parent (<em>super</em>) module into the current <code>tests</code> module. So instead of calling <code>super::words()</code>, we can simply call <code>words()</code>.</p>
<p>Remember that in Rust, a <em>private</em> function can be accessed <em>by the parent module, and by all of the parent's submodules</em>. Because of that rule, we can import the <code>words</code> function, which is private, into the <code>tests</code> module.</p>
<p>At this point, we've created our testing module and imported the functions we want to test. Let's look at the tests.</p>
<h2>Test Functions</h2>
<p>Here's the first test function:</p>
<pre class="highlight rust"><code><span class="err">#</span><span class="p">[</span><span class="n">test</span><span class="p">]</span>
<span class="k">fn</span> <span class="nf">test_words</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">input</span> <span class="o">=</span> <span class="s">"this is the way the world ends"</span><span class="p">;</span>
<span class="k">let</span> <span class="n">expect</span> <span class="o">=</span> <span class="nd">vec!</span><span class="p">[</span><span class="s">"this"</span><span class="p">,</span> <span class="s">"is"</span><span class="p">,</span> <span class="s">"the"</span><span class="p">,</span> <span class="s">"way"</span><span class="p">,</span> <span class="s">"the"</span><span class="p">,</span> <span class="s">"world"</span><span class="p">,</span> <span class="s">"ends"</span><span class="p">];</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="nf">words</span><span class="p">(</span><span class="n">input</span><span class="p">)</span><span class="py">.collect</span><span class="p">::</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><&</span><span class="nb">str</span><span class="o">>></span><span class="p">(),</span> <span class="n">expect</span><span class="p">)</span>
<span class="p">}</span>
</code></pre>
<p>My Go naming habits have me prefixing the test function with <code>test_</code>, but as far as I can tell, that's not an idiom in Rust. Maybe calling it <code>words_equal</code> would have been just as acceptable.</p>
<p>But to make this function a test, we need to prefix it with the <code>#[test]</code> attribute. This is what indicates that the function a testing target. If I omit this attribute, I'll see an error:</p>
<pre class="highlight plaintext"><code>warning: function is never used: `test_words`
--> src/lib.rs:18:5
|
18 | fn test_words() {
| ^^^^^^^^^^^^^^^
|
= note: #[warn(dead_code)] on by default
</code></pre>
<p>Inside of a test module I can add utilities, mocks, etc. and as long as I don't label them with <code>#[test]</code> they will not be mistaken for tests.</p>
<p>In the four test functions I created, I used an assertion macro (<code>assert_eq!</code>). There are actually four macros that are useful for testing:</p>
<ul>
<li><code>assert!</code>, which asserts that the value is <code>true</code></li>
<li><code>assert_eq!</code>, which asserts that two values are equal</li>
<li><code>assert_ne!</code>, which asserts that two values are not equal</li>
<li><code>panic!</code> which causes the test to fail</li>
</ul>
<blockquote>
<p>Go has two classes of failure: <code>t.Error</code> and <code>t.Fatal</code>. Standard Rust only has one type of failure.</p>
</blockquote>
<p>There's also a <code>#[should_panic]</code> annotation. Decorate a function with this to indicate that a test is considered passing <em>if and only if</em> it panics.</p>
<p>From here, our tests are straightforward. We merely compare the output of our two functions to the expected results.</p>
<h2>Breaking Out Tests into a Separate File</h2>
<p>To be honest, I'm not totally sure what the accepted idioms are for breaking tests out into separate files, but it turns out that it is relatively easy to do.</p>
<p>Modules can be <a href="https://doc.rust-lang.org/rust-by-example/mod/split.html">split into separate files</a>, and
<a href="https://doc.rust-lang.org/book/second-edition/ch11-01-writing-tests.html">tests are organized into modules</a>. So we can split tests into a separate file like this.</p>
<p>First, here's the main file for the library in <code>lib.rs</code>:</p>
<pre class="highlight rust"><code><span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="nn">str</span><span class="p">::</span><span class="n">SplitWhitespace</span><span class="p">;</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="nf">initials</span><span class="p">(</span><span class="n">phrase</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">)</span> <span class="k">-></span> <span class="nb">String</span> <span class="p">{</span>
<span class="nf">words</span><span class="p">(</span><span class="n">phrase</span><span class="p">)</span><span class="nf">.map</span><span class="p">(</span>
<span class="p">|</span><span class="n">word</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">|</span> <span class="n">word</span><span class="nf">.chars</span><span class="p">()</span><span class="nf">.next</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">()</span>
<span class="p">)</span><span class="py">.collect</span><span class="p">::</span><span class="o"><</span><span class="nb">String</span><span class="o">></span><span class="p">()</span><span class="nf">.to_uppercase</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">words</span><span class="p">(</span><span class="n">phrase</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">)</span> <span class="k">-></span> <span class="n">SplitWhitespace</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">phrase</span><span class="nf">.split_whitespace</span><span class="p">()</span>
<span class="p">}</span>
<span class="cp">#[cfg(test)]</span>
<span class="k">mod</span> <span class="n">tests</span><span class="p">;</span>
</code></pre>
<p>On the last two lines of that file, we declare a testing module, but we don't put anything in the module. This will cause the compiler to <a href="https://doc.rust-lang.org/rust-by-example/mod/split.html">look for tests.rs</a>. We can oblige it by putting all the tests in <code>tests.rs</code>:</p>
<pre class="highlight rust"><code><span class="k">use</span> <span class="nn">super</span><span class="p">::</span><span class="o">*</span><span class="p">;</span>
<span class="cp">#[test]</span>
<span class="k">fn</span> <span class="nf">test_words</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">input</span> <span class="o">=</span> <span class="s">"this is the way</span><span class="se">\n</span><span class="s"> the world ends"</span><span class="p">;</span>
<span class="k">let</span> <span class="n">expect</span> <span class="o">=</span> <span class="nd">vec!</span><span class="p">[</span><span class="s">"this"</span><span class="p">,</span> <span class="s">"is"</span><span class="p">,</span> <span class="s">"the"</span><span class="p">,</span> <span class="s">"way"</span><span class="p">,</span> <span class="s">"the"</span><span class="p">,</span> <span class="s">"world"</span><span class="p">,</span> <span class="s">"ends"</span><span class="p">];</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="nf">words</span><span class="p">(</span><span class="n">input</span><span class="p">)</span><span class="py">.collect</span><span class="p">::</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><&</span><span class="nb">str</span><span class="o">>></span><span class="p">(),</span> <span class="n">expect</span><span class="p">)</span>
<span class="p">}</span>
<span class="cp">#[test]</span>
<span class="k">fn</span> <span class="nf">test_initials</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">input</span> <span class="o">=</span> <span class="s">"not with a bang but a whimper"</span><span class="p">;</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="nf">initials</span><span class="p">(</span><span class="n">input</span><span class="p">),</span> <span class="s">"NWABBAW"</span><span class="p">);</span>
<span class="p">}</span>
<span class="cp">#[test]</span>
<span class="k">fn</span> <span class="nf">empty_words</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">input</span> <span class="o">=</span> <span class="s">""</span><span class="p">;</span>
<span class="k">let</span> <span class="n">expect</span><span class="p">:</span> <span class="nb">Vec</span><span class="o"><&</span><span class="nb">str</span><span class="o">></span> <span class="o">=</span> <span class="nn">Vec</span><span class="p">::</span><span class="nf">new</span><span class="p">();</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="nf">words</span><span class="p">(</span><span class="n">input</span><span class="p">)</span><span class="py">.collect</span><span class="p">::</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><&</span><span class="nb">str</span><span class="o">>></span><span class="p">(),</span> <span class="n">expect</span><span class="p">);</span>
<span class="p">}</span>
<span class="cp">#[test]</span>
<span class="k">fn</span> <span class="nf">empty_initials</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">input</span> <span class="o">=</span> <span class="s">""</span><span class="p">;</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="nf">initials</span><span class="p">(</span><span class="n">input</span><span class="p">),</span> <span class="s">""</span><span class="p">);</span>
<span class="p">}</span>
</code></pre>
<p>Now if we run <code>cargo test</code>, we'll see the usual testing output:</p>
<pre class="highlight shell_session"><code><span class="w">$ </span><span class="nc">cargo</span><span class="kv"> test
</span> Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
Running target/debug/deps/wordutils-9a757f9d84faff12
running 4 tests
test tests::empty_words ... ok
test tests::empty_initials ... ok
test tests::test_initials ... ok
test tests::test_words ... ok
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Doc-tests wordutils
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
</code></pre>
<p>Again, I'm not sure if this is the preferred way to write tests, but it seems like a nice way to organize things.</p>
<h2>Specifying the Test to Run</h2>
<p>Here's one final note on running tests with <code>cargo</code>. In Go, we can select specific tests to run by regular expression: <code>go test -run REGEXP</code>. With Cargo, you can call specific tests by name:</p>
<pre class="highlight shell_session"><code><span class="w">$ </span><span class="nc">cargo</span><span class="kv"> test empty_words
</span></code></pre>
<p>There are several flags (like <code>--exclude</code>) that can impact which tests are run. But as far as I know, there's no equivalent regular expression version.</p>
<h2>Conclusion</h2>
<p>The focus of this article has been on how to unit test Rust code, compared to Go. As a Rust neophyte, I have been pleasantly surprised by the similarities. The things I love about Go's testing are also present in Rust's testing. And I find Rust's built-in assertions, module-based tests, and attribute annotations to be elegant.</p>
<p>But unit testing is only one of the kinds of tests we care about. In the next installment of the series, we will see how Rust's testing stands up against Go's when it comes to benchmarks, documentation tests, and functional tests.</p>
Be Nice And Write Stable Codehttp://technosophos.com/2018/07/04/be-nice-and-write-stable-code.html2018-07-04T15:13:00+00:002018-07-06T16:28:41+00:00Article Author<p>Stop rearchitecting your code! The professional developer values stability over "code purity." Instead of pursuing a Shangra-La vision of code perfection with each and every release, just be nice and write <em>stable APIs</em>. In this post, I talk about taking practical steps toward writing code that remains stable over time.</p>
<h2>The Non-Goal</h2>
<p>It is completely unhelpful to begin with a suggestion like this:</p>
<blockquote>
<p>When you write code, make it future-proof.</p>
</blockquote>
<p>If history teaches us anything, it's that we are lousy predictors of what the future holds. <em>Of course</em> it is a good practice to strive toward clean APIs, flexible design, and thoughtful defaults. But inevitably, your users and emerging requirements will surprise you.</p>
<p>What I want to talk about is how to best deal with those surprises, and how to avoid introducing surprises (and frustration) to those who use your code.</p>
<h2>Versions and SemVer</h2>
<p>In software, we use version numbers to signal that something has changed. Version numbering schemes go from dead simple (integers that increment with every release, or date stamps) to surprisingly complex (<code>1.0~pre3+dfsg-0.1+b2</code>, <code>2.1.1+git20160721~8efc468-2</code>, and <code>1.2.0+LibO5.2.7-1+deb9u4</code> are a few versions spotted in the wild).</p>
<p>But when it comes to software version numbers, the current leader in version numbering schemes is SemVer (or Semantic Versioning). Don't be fooled, though! Many people claim to know how SemVer works, but have never read <a href="https://semver.org/">the specification</a>. Since this is a critical piece of what we are about to talk about, here is a summary of the spec:</p>
<p>Version numbers take the form <code>X.Y.Z</code>, sometimes augmented with additional pre-release and build information: <code>X.Y.Z-AAA#BBB</code>. And each of those fields means something <em>well defined and specific</em>.</p>
<ul>
<li><code>X</code> is the <em>major</em> number. Changes in this indicate <em>breaking changes</em> to the API (and/or behavior).</li>
<li><code>Y</code> is the <em>minor</em> number. Changes to this number indicate that new features were added, but that no APIs are broken as a result.</li>
<li><code>Z</code> is the <em>patch</em> version. Changes to this indicate that internal changes were made, but that no changes (even compatible changes) were made to the API.</li>
</ul>
<p>These three are the important ones for us. Again, I suggest taking 15 minutes to read the entire spec.</p>
<p>Countless projects use a format that looks like SemVer, but many of them ignore the semantics behind the version number. Often, it seems that version numbers are incremented by "gut feel" instead of any consistent semantic: "This feels like a minor version update."</p>
<p>The intention of this post is to explain how to write software in a way that actually adheres to semantic versioning. Get rid of "gut feel" version numbers and give your users some peace of mind.</p>
<h3>But Why?</h3>
<p>Why bother using a semantic versioning scheme? What's wrong with just updating numbers arbitrarily? The reason is simple: Version numbers help your users understand something about the nature of the changes they can expect. If you don't follow a pattern, they are left guessing. And this frustrates people.</p>
<p>Following SemVer introduces rigor on two fronts:</p>
<ol>
<li>It sends clear signals to users about the depth of changes they can expect in a release.</li>
<li>It sends a clear signal to your developers about what is, and what is not, allowed when it comes to changing the code.</li>
</ol>
<p>I cannot understate the importance of (2). SemVer helps us impose self-discipline, which in turn minimizes both internal and external disruption.</p>
<h2>Patterns of Change</h2>
<p>With the SemVer discussion out of the way, we can now talk about the actual patterns of change.</p>
<p>Remember, the usability of code is a focal point of the professional software developer. Predictable patterns of change are a boon to usability.</p>
<h3>Reorganizing, Refactoring, and Renaming</h3>
<p>There is no clearer way to state the point than this: If you reorganize the package structure of your public API, or if you do a major renaming, or if you choose to change the methods/structs/classes/etc of your public API, <em>you must increment the major version number</em>.</p>
<p>That's it. There is no grey area here. Such changes mean that anyone who's using your code will experience breakage.</p>
<p>When it comes to working on minor updates, dealing with this means exercising discipline. Yes, the package structure might be poor. Yes, the code might be ugly. But you must wait until the right moment to fix that.</p>
<p>Of course, it's okay to make internal changes that don't touch any public API items. So minor internal-only refactoring can be done in minor, and even patch, releases (though we don't recommend doing it in patch releases).</p>
<blockquote>
<p><strong>Note:</strong> Stop trying to justify your refactoring with the "public but internal" argument. <strong>If the language spec says it's public, it's public.</strong> Your intentions have nothing to do with it.</p>
</blockquote>
<p>So in effect, the following are not be be changed except during major updates:</p>
<ul>
<li>Package structure</li>
<li>Public class, struct, enum, trait, interface, etc. names, nor the names of any of the items on these</li>
<li>Constants or public variable names or values</li>
<li>Function/method names</li>
<li>Function/method signatures for existing functions <em>except</em> where the change is additive and the added argument is optional. Return value types and exceptions must also not change.</li>
</ul>
<p>The bottom line: Refactoring, renaming, and reorganizing is a sweet temptation. But this is a temptation that must be resisted when doing a minor/patch releases. Part of being a professional software developer is creatively coping with imperfect code.</p>
<p>But, you might be saying, how do you add new features without changing any of these? That is the subject of the next few sections.</p>
<h3>Introducing New Features</h3>
<p>Minor versions may introduce new features, but features must be introduced without breaking existing APIs.</p>
<p>Features are additive in nature: They bring new things, but do not modify or delete existing things. To that end, these are safe as part of a feature release:</p>
<ul>
<li>Adding a field or method to a struct/class/enum/etc.</li>
<li>Adding a new struct/class/enum/etc. or adding new variables, constants, functions, packages, etc.</li>
<li>Adding new configuration options (but see the next section)</li>
<li>Making something that was non-public into something that is public (e.g. exposing a private API as a public API)</li>
</ul>
<p>However, there are a few changes that are sometimes done under the guise of a feature, but which are breaking changes that must be avoided:</p>
<ul>
<li>Changing values of constants, variables, etc.</li>
<li>Changing a function or method <em>signature</em> (e.g. adding more params or changing the return type)
<ul>
<li>There is an exception here if a language supports adding <em>optional</em> parameters in a way that will still make old calls to the function to work exactly the same.</li>
</ul></li>
<li>Changing an item from public-scoped to non-public (hiding an API)</li>
</ul>
<h3>Modifying by Adding Alternatives</h3>
<p>Consider the case where you begin with code like this:</p>
<pre class="highlight go"><code><span class="k">func</span><span class="x"> </span><span class="n">ListItems</span><span class="p">(</span><span class="n">query</span><span class="x"> </span><span class="n">Query</span><span class="p">)</span><span class="x"> </span><span class="n">Items</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="c">// Code to fetch and list the items</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre>
<p>The code above might, for example, do a database query and fetch all of the results.</p>
<p>Now a feature request rolls in: "We need to add paging to the list functions." The temptation is to do this:</p>
<pre class="highlight go"><code><span class="k">func</span><span class="x"> </span><span class="n">ListItems</span><span class="p">(</span><span class="n">query</span><span class="x"> </span><span class="n">Query</span><span class="p">,</span><span class="x"> </span><span class="n">limit</span><span class="x"> </span><span class="kt">int</span><span class="p">,</span><span class="x"> </span><span class="n">offset</span><span class="x"> </span><span class="kt">int</span><span class="p">)</span><span class="x"> </span><span class="n">Items</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="c">// ...</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre>
<p>But that is an API breaking change. The correct way to handle this is to introduce a new function:</p>
<pre class="highlight go"><code><span class="k">func</span><span class="x"> </span><span class="n">ListItems</span><span class="p">(</span><span class="n">query</span><span class="x"> </span><span class="n">Query</span><span class="p">)</span><span class="x"> </span><span class="n">Items</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">ListItemsWithLimit</span><span class="p">(</span><span class="n">query</span><span class="p">,</span><span class="x"> </span><span class="m">0</span><span class="p">,</span><span class="x"> </span><span class="m">0</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">ListItemsWithLimit</span><span class="p">(</span><span class="n">query</span><span class="x"> </span><span class="n">Query</span><span class="p">,</span><span class="x"> </span><span class="n">limit</span><span class="x"> </span><span class="kt">int</span><span class="p">,</span><span class="x"> </span><span class="n">offset</span><span class="x"> </span><span class="kt">int</span><span class="p">)</span><span class="x"> </span><span class="n">Items</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="c">// ...</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre>
<p>Note that we have adjusted the internals on the old to replicate the exact behavior of the old function, but by calling into the new function.</p>
<p><em>It is fine to mark functions as deprecated when you do this.</em> In fact, this is why languages like Java cleverly added built-in support for deprecation. Professional-grade development involves a strategy of deprecation with <em>eventual</em> removal, even if that removal is years down the road.</p>
<p>Importantly, if the newly introduced function cannot replicate the old feature set, you are obligated (unless security concerns dictate otherwise) to provide the old API's functionality to the greatest extent possible.</p>
<h4>Beware The Dubious Work-Around</h4>
<p>There is an accepted pattern that works well for many things, but which some developers employ to "get around" the SemVer constraints on modifying function signatures:</p>
<pre class="highlight plaintext"><code>func ListItems(query Query, options Map) Items {
// ...
}
</code></pre>
<p>In this pattern, the <code>options</code> map is an arbitrary set of key/value options. This pattern itself is fine <em>unless</em> the default behavior changes when a new option is introduced. When adding a new option, the professional developer ensures that when that option is not present, the code behaves the same as it did in the last release.</p>
<h3>Deprecating</h3>
<p>We touched on deprecation above. But I want to summarize the deprecation strategy:</p>
<ol>
<li>Mark a thing as deprecated <em>as soon as it is considered deprecated</em>, even if that is a patch or minor release. Deprecation, after all, is a <em>warning</em> condition, not an <em>error</em> condition.</li>
<li>Do not change the behavior of the deprecated thing during minor or patch releases</li>
<li>Remove deprecated things only at major version changes. Until that time, you're still on the hook for supporting them.</li>
</ol>
<p>Deprecation is a <em>signal</em> that in the future a thing will be removed. But it is not an excuse to change, delete, or ignore the functionality of that bit of code outside of the SemVer constraints.</p>
<h3>Errors and Exceptions</h3>
<p>One of the most frustrating outages I ever experienced occurred because of a seemingly innocuous change to an upstream library: During a minor release update, the library changed the exception type that it threw on a particular error.</p>
<p>One of the functions in the library threw an <code>IOException</code> whenever a network error occurred, and other exceptions for other problems.</p>
<p>We used the throwing of an <code>IOException</code> to kick off our retry logic. Given that network failures were a frequent occurrence for our particular conditions, this was an important feature.</p>
<p>But during a minor version change, the developers decided to simplify the API by catching all of the different exceptions (including the <code>IOException</code>) and wrap them in a single generic exception. (Incidentally, the API itself did not change because it was something like <code>func Read(in Reader) error</code>, where <code>error</code> was a parent of all exceptions).</p>
<p>When we upgraded, all our tests passed (because our test fixtures emulated the old behavior and our network was not unstable enough to trigger bad conditions), and production rolled out just fine. But our customers began complaining that the product was much less reliable. Why? Because the retry logic was never triggered. So our app suddenly was as unstable as the network it was on.</p>
<p>The bottom line: Even error handling is part of your public API.</p>
<h3>Resisting Subtle Changes</h3>
<p>Sometimes subtle but ill-planned changes can cause major breakages for your users.</p>
<p>Here's a short story of how a trivial change in one of our dependencies led to a series of production catastrophes for our users:</p>
<p>We depended on a library that provided a client/server RPC-like protocol. This library had long been marked stable, and indeed stability is one of the touted features of this library. But the developers introduced a very subtle change that <em>appeared</em> to follow the stability requirements, but which actually introduced a serious compatibility flaw. The change went something like this:</p>
<p>The library allowed us to set a maximum message size. The default was 256k, but we wanted it to be significantly larger. So we set this option:</p>
<pre class="highlight go"><code><span class="n">config</span><span class="o">.</span><span class="n">MaxMesageSize</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="m">1024</span><span class="x">
</span></code></pre>
<p>This made it possible for client and server to send each other messages up to 1M. But at some point, a minor release of the package made a very subtle, but killer, change: They split upstream and downstream message sizes into two variables. And here's the killer: They did this by merely creating a second variable (<code>config.MaxInboundMessageSize</code>) and changing the behavior of the first to impact only outbound message size.</p>
<p>When we upgraded the package, all seemed to work well. Code compiled. Tests passed. Early users saw no problems. Then we shipped our new version with this updated dependency. And suddenly angry users started filing issues. Stuff that worked yesterday was broken today.</p>
<p>Why? Because behind the scenes, the inbound message size had dropped from 1M to it's default 256k. And while nothing in our early testing sent messages larger than 256k, there were plenty of production instances that did.</p>
<p>The upstream library maintainers had introduced a serious bug into our code by silently changing the behavior of their code, even though they didn't (in a pedantic sense) "break" SemVer.</p>
<p>What should they have done?</p>
<p>I would argue that breaking the size limits into an inbound and an outbound is a totally legitimate thing to do during a minor release. It was just done wrong.</p>
<p>The right way to address a configuration change like this is to introduce <em>two new variables</em> and then add default support for the old one.</p>
<p>Thus, in practice, it would look like this:</p>
<pre class="highlight go"><code><span class="k">type</span><span class="x"> </span><span class="n">Config</span><span class="x"> </span><span class="k">struct</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="c">// Old one</span><span class="x">
</span><span class="n">MaxMessageSize</span><span class="x"> </span><span class="kt">int</span><span class="x">
</span><span class="c">// New settings</span><span class="x">
</span><span class="n">MaxInboundMessageSize</span><span class="x"> </span><span class="kt">int</span><span class="x">
</span><span class="n">MaxOutboundMessageSize</span><span class="x"> </span><span class="kt">int</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre>
<p>And then the internal code in the library would handle the legacy case while still offering the new functionality:</p>
<pre class="highlight go"><code><span class="c">// Support clients who set the old config</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">config</span><span class="o">.</span><span class="n">MaxMessageSize</span><span class="x"> </span><span class="o">></span><span class="x"> </span><span class="m">0</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">config</span><span class="o">.</span><span class="n">MaxInboundMessageSize</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">config</span><span class="o">.</span><span class="n">MaxMessageSize</span><span class="x">
</span><span class="n">config</span><span class="o">.</span><span class="n">MaxOutboundMessageSize</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">config</span><span class="o">.</span><span class="n">MaxMessageSize</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre>
<p>With something like this in the base library, any tools that use it would get the old behavior if they used the old configuration param, but could opt into using the newer options instead.</p>
<h3>Bugs and Security Fixes: How To Handle Real Life</h3>
<p>Guidance that you should never change certain things is all well and good until reality comes a-callin'. But what if that public constant or variable introduces a security issue, or causes the server to crash?</p>
<p>When the real world comes crashing in, we make exceptions. But professional software developers make them wisely and carefully.</p>
<p>The important concept here is the <em>minimally invasive change</em>. That is, when patching bugs or security releases, we may need to change the API, but we should do it by changing the absolute minimum number of things we can get away with. And we do that <em>even if it means sacrificing our "architectural purity"</em>.</p>
<p>I will plead guilty for introducing a global variable as a stop-gap to fix the internals of a function without changing the function signature. It was ugly. I was ashamed. But it ensured backward compatibility, and that was the important thing. If I had changed the function call, thousands of users would have had to change their code. But with the ugly global, only the few who needed to tune that particular parameter were impacted (and only by having the option to set something they could not previously control).</p>
<p>But a security issue or major bug is a legitimate reason to change things like default values or even larger macro behaviors. If the change is big enough, you're still obligated to change the major version of your code. SemVer doesn't give a free pass on that, and failing to do so still undermines user confidence.</p>
<p>But for less intrusive changes, I personally feel like you can make some minor SemVer transgressions provided:</p>
<ul>
<li><p>You make this <em>very</em> clear in your release notes. </p>
<p>"The value of MaxBufferSize was adjusted downward to 2048 because we discovered a buffer overflow in a lower level library for any larger buffer size. See issue #4144"</p></li>
<li><p>The code is clearly commented:</p></li>
</ul>
<pre class="highlight plaintext"><code>// MaxBufferSize sets the maximum size of the network buffer.
// Prior to version 2.5.1, this was 4096. Due to a security flaw
// reported in #4144 that resulted in a buffer overflow, we
// lowered this to 2048.
MaxBufferSize = 2048
</code></pre>
<h2>Conclusion</h2>
<p>The professional software developer has long-term usability and stability as a goal. Yes, well-architected code is important. But there is a time and place for making that your focus. And maintenance releases (minor and patch versions) are not an occasion to refactor, re-organize, or make sweeping modifications.</p>
<p>Be conscientious about how much effort the users of your code put into <em>using</em> your code. I can tell you from experience what we do when the maintenance burden you impose on us gets wearying: We stop using your tools (or we fork them).</p>
<p>SemVer is a communications tool. But to use it well, we must use it accurately. And that means writing code focused on stability.</p>