From Go To Rust - Advanced Testing

Jul 25 2018

For the fifth installment of this series, we'll take a look at benchmarking, documentation testing, and integration testing. As usual, we'll start with an example in Go and see how it translates to Rust. Along the way, we'll be learning about the Rust language.

If you want to catch up on the series:

As I compared Go and Rust in the last post, I noted that Go has support for testing beyond mere unit tests. And that is where we'll start today.

Go Goes Beyond Unit Tests

Go's built-in testing package defines three classes of tests:

  • Unit tests, which we looked at last week
  • Benchmarks for testing performance of your code
  • Documentation functions, which can be tested automatically

So let's kick things off with an example of benchmarking and documentation functions. We'll continue where we left off last time. Here's the base library that we are writing tests for.

wordutils.go:

package wordutils

import (
    "bufio"
    "strings"
)

// Initials returns a string with the first letter of each word in the given string.
func Initials(phrase string) (string, error) {
    wrds, err := words(phrase)
    if err != nil {
        return "", err
    }
    initials := ""
    for _, word := range wrds {
        initials += word[0:1]
    }
    return strings.ToUpper(initials), nil
}

func words(str string) ([]string, error) {
    wordList := []string{}
    scanner := bufio.NewScanner(strings.NewReader(str))
    scanner.Split(bufio.ScanWords)
    for scanner.Scan() {
        wordList = append(wordList, scanner.Text())
    }
    return wordList, scanner.Err()
}

I am not going to reproduce the unit tests we covered last week. Instead, we are going to dive right into the new testing material, which will also be in wordutils_test.go.

package wordutils

import (
    "fmt"
    "testing"
)

func BenchmarkInitials(b *testing.B) {
    text := "I have measured my life in coffee spoons"
    for i := 0; i < b.N; i++ {
        if _, err := Initials(text); err != nil {
            panic(err)
        }
    }
}

func ExampleInitials() {
    text := "J. Alfred Prufrock"
    out, err := Initials(text)
    if err != nil {
        panic(err)
    }
    fmt.Print(out)
    // Output:
    // JAP
}

The BenchmarkInitials function will be called when we execute go test -bench. It will run the test multiple times until suitable benchmarks can be generated:

$ go test --bench .
goos: darwin
goarch: amd64
pkg: github.com/technosophos/wordutils
BenchmarkInitials-4       500000          2286 ns/op
PASS
ok      github.com/technosophos/wordutils   1.180s

The ExampleInitials function is part of the documentation. So if we run godoc on our library, we will see the example: godoc -html github.com/technosophos/wordutils Initials. (Unfortunately, the examples are not printed in the plain-text version of godoc help.)

But to make sure that our examples stay current and accurate, Go will automatically run them as tests during regular unit testing:

$ go test -v .
=== RUN   TestWords
--- PASS: TestWords (0.00s)
=== RUN   TestInitials
--- PASS: TestInitials (0.00s)
=== RUN   ExampleInitials
--- PASS: ExampleInitials (0.00s)
PASS
ok      github.com/technosophos/wordutils   (cached)

As with unit tests, Go determines the kind of test based on the function signature.

  • BenchmarkXXX(b *testing.B) is a benchmark
  • ExampleXXX() is an example

There are a few other variations of these patterns that you can use, but the basic idea is that the testing tool reflects over the code to determine what to execute during a testing cycle.

As we'll see with Rust, there are four supported classes of tests:

  • Unit tests (again, covered last time)
  • Benchmarks, which are new and still marked unstable
  • Documentation examples
  • Integration tests

Rust Benchmarks

Rust is introducing benchmark testing. It is available in the unstable builds of Rust, but not yet in the official stable build.

Enabling Benchmarking

So to test this out, we need to enable unstable features. Inside of the wordutils project, we need to run rustup override add nightly to switch us over to using the nightly build for this particular project.

$ rustup update nightly
$ rustup override add nightly
info: using existing install for 'nightly-x86_64-apple-darwin'
info: override toolchain for '/Users/mbutcher/Code/Rust/wordutils' set to 'nightly-x86_64-apple-darwin'

  nightly-x86_64-apple-darwin unchanged - rustc 1.29.0-nightly (6a1c0637c 2018-07-23)

Now we can use the benchmarking features.

Writing Benchmark Tests

Last time we created a wordutils library with Cargo. By the end, we were experimenting with several features of package organization. But let's start off with a simplified version of the code we used last time, and add just the benchmark.

#![feature(test)]
extern crate test;

pub fn initials(phrase: &str) -> String {
    phrase.split_whitespace().map(
        |word: &str| word.chars().next().unwrap()
    ).collect::<String>().to_uppercase()
}

#[cfg(test)]
mod tests {
    use super::*;
    use test::Bencher;

    #[bench]
    fn bench_initials(b: &mut Bencher) {
    let input = "J. Alfred Prufrock";
        b.iter(|| initials(input));
    }
}

Since we are using an unstable feature, we need to tell Rust explicitly that we know what we're doing when we use the test module:

#![feature(test)]
extern crate test;

By enabling the test feature, we indicate that we are using features that would otherwise be disabled because of stability flags.

The stability mechanism of Rust is a cool way of gradually introducing features, while letting people like us kick the tires and report errors.

Inside of our own mod test, we add just one benchmarking test:

use test::Bencher;

#[bench]
fn bench_initials(b: &mut Bencher) {
    let input = "I have measured my life in coffee spoons";
    b.iter(|| initials(input));
}

Instead of using the #[test] attribute, we use #[bench] to indicate that this is a benchmark. (Unlike Go, Rust function names have no impact on whether this is considered a benchmark test.)

The Bencher is Rust's equivalent of testing.B in Go. We've seen in previous posts how Rust uses iterators and anonymous functions. And in the example above, Rust is doing basically the same thing that Go does, only more compactly.

In Go, we wrote a benchmark like this:

text := "I have measured my life in coffee spoons"
for i := 0; i < b.N; i++ {
    if _, err := Initials(text); err != nil {
        panic(err)
    }
}

Looking at the for loop, we can see that we ran the test as many times as b.N indicates. But we did have to explicitly create the for loop for this.

Conceptually, Rust is doing the same thing. It is running the || initials(input) test as many times as bench.iter() dictates.

Recall from previous posts that |params| body is the syntax for Rust closures. bench.iter() takes a closure with zero parameters.

Now we can run the benchmark with cargo bench:

$ cargo bench
   Compiling wordutils v0.1.0 (file:///Users/mbutcher/Code/Rust/wordutils)
    Finished release [optimized] target(s) in 2.33s
     Running target/release/deps/wordutils-0770f6e0ea5ebd16

running 1 test
test tests::bench_initials ... bench:         512 ns/iter (+/- 53)

test result: ok. 0 passed; 0 failed; 0 ignored; 1 measured; 0 filtered out

What we see from the result is that our initials test took about 512ns per operation, with a variance of 53. (I'm actually a little surprised. This is 4x faster than my Go implementation.)

As benchmarking is still fairly new, to learn more you'll need to take a peek at the nightly documentation on this feature.

Documentation Testing

While benchmarking is still a new feature of Rust, writing and testing examples is a stable feature.

I'm not going to beat around the bush about this, but I think Rust's documentation testing feature is a thing of beauty. Why? Because examples are embedded in the documentation blocks, and executed automatically. When I'm in the process of writing code, I feel like this is more amenable to my coding practice.

My biggest complaint with Go examples is that because they require a context switch, a special method signature, and special markup at the end, they are weird to write. Most Go developers simply don't write examples this way. Also, because Go's example methodology is a biased toward string testing, it's hard to make these examples reflective of real usage.

Rust goes the other direction: Examples are written as part of source code comments, and are executed almost like mini-programs.

But to understand how this testing works, we need to spend a moment on Rust source code documentation.

Documenting Rust

Like Go (and many other languages), Rust supports extracting documentation from the source code. Rust uses Markdown as the documentation format, which means we can write docs that are a little richer than Go's when it comes to formatting.

Let's document our initials() function:

/// Given a string, extract the initials.
/// 
/// Initials are composed of the first letter of each word, capitalized.
/// They are then joined together with no spaces.
pub fn initials(phrase: &str) -> String {
    phrase.split_whitespace().map(
        |word: &str| word.chars().next().unwrap()
    ).collect::<String>().to_uppercase()
}

Comments use three slashes (///). The first line is a complete, punctuated sentence. Unlike Go documentation, this does not begin with the name of the item being documented.

After a line break, we can add a more complete description. Then, to generate the documentation, we run cargo docs. The resulting documentation will be written into the target/doc/wordutils directory as HTML. We'll see an example shortly.

Adding an example in Markdown

The idea of Rust's example documentation is that it should accurately replicate how the function would be called in context. So we just embed a snippet of code right into the Markdown, using the sort of conventions you would normally use:

/// Given a string, extract the initials.
/// 
/// Initials are composed of the first letter of each word, capitalized.
/// They are then joined together with no spaces.
/// 
/// # Example
/// 
/// ```rust
/// let out = wordutils::initials("hello beautiful world");
/// assert_eq!(out, "HBW");
/// ```
pub fn initials(phrase: &str) -> String {
    phrase.split_whitespace().map(
        |word: &str| word.chars().next().unwrap()
    ).collect::<String>().to_uppercase()
}

Unlike our unit tests, we do need to call the package by its full name (or use a use).

Rust will run the inlined examples as unit tests during the testing phase:

$ cargo test
  Compiling wordutils v0.1.0 (file:///Users/mbutcher/Code/Rust/wordutils)
    Finished dev [unoptimized + debuginfo] target(s) in 1.07s
     Running target/debug/deps/wordutils-e01fe756921f1114

running 1 test
test tests::bench_initials ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests wordutils

running 1 test
test src/lib.rs - initials (line 11) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

In the Doc-tests section, we can see that it ran our documentation test. (Interestingly, it also appears to have run bench_intials to make sure that it didn't fail.)

A documentation test is considered successful if it doesn't panic. (It does not, however, check whether one knows where one's towel is.)

You can also show examples of failures by annotating the markdown code block with should_panic. This test is totally contrived, but shows the basic idea:

/// Given a string, extract the initials.
/// 
/// Initials are composed of the first letter of each word, capitalized.
/// They are then joined together with no spaces.
/// 
/// # Example
/// 
/// ```rust
/// let out = wordutils::initials("hello beautiful world");
/// assert_eq!(out, "HBW");
/// ```
/// 
/// # Panics
/// 
/// ```rust,should_panic
/// let out = wordutils::initials("");
/// assert_eq!(out, "hello");
/// ```
pub fn initials(phrase: &str) -> String {
    phrase.split_whitespace().map(
        |word: &str| word.chars().next().unwrap()
    ).collect::<String>().to_uppercase()
}

In the second example, the assert_eq! will panic because the initials of "" will not be "hello". But when we run the test, it will succeed because it was expecting that example to panic.

$ cargo test
   Compiling wordutils v0.1.0 (file:///Users/mbutcher/Code/Rust/wordutils)
    Finished dev [unoptimized + debuginfo] target(s) in 2.55s
     Running target/debug/deps/wordutils-e01fe756921f1114

running 1 test
test tests::bench_initials ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests wordutils

running 2 tests
test src/lib.rs - initials (line 11) ... ok
test src/lib.rs - initials (line 18) ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Now if we generate the documentation with cargo doc, this is what it will look like:

Cargo Docs

At this point, we've seen both benchmarking and documentation testing. And as with Go, we haven't explored every nook and cranny of the framework, but we've gotten the basic idea.

However, we have one more thing to cover. And this is a feature that Go doesn't support in its core toolset: Integration testing.

Integration Tests in Rust

The last category of tests to cover is integration tests. Typically, while unit tests cover individual functions and are "close to the source", integration tests are designed to show that, from an outside perspective, things work as advertised.

Often, unit tests will use fixtures and mocks to test just very specific parts of the code. Integration testing more often forgoes mocks (at least mocks of internal things), and tests that the code is functioning together as a whole.

Rust has a top-level concept of integration tests. Inside of your Cargo project, they are placed in the tests/ directory adjacent to src/ and target/:

.
├── Cargo.lock
├── Cargo.toml
├── src
│   ├── lib.rs
│   └── tests.rs
├── target
│   ├── debug
│   ├── doc
│   └── release
└── tests
    └── integration_tests.rs

Integration tests are constructed the way the unit tests and documentation tests are: Write some code, use some asserts to make sure it does what it is supposed to.

Unfortunately for us, we don't have a whole lot of "integration" to do. But we'll still see the main pattern for integration tests, and see how this differs from unit tests.

I created an integration test in tests/ named integration_tests.rs. Zero points for originality.

extern crate wordutils;

use wordutils::initials;

#[test]
fn do_initials() {
    assert_eq!(initials("j. alfred prufrock"), "JAP");
}

The main thing to notice about this is that integration tests are structured the same way that an external tool would use the library.

Pro Tip: That means that the contents of a crate's tests/ directory is a great place to figure out how to use a library.

So we use extern to declare that we are using wordutils and we use use to import the initials function into our current namespace.

However, we still have to use the #[test] attribute to declare that do_initials() is a test function.

As with unit and documentation tests, Cargo will do us the favor of running these tests as part of cargo test:

$ cargo test
   Compiling wordutils v0.1.0 (file:///Users/mbutcher/Code/Rust/wordutils)
    Finished dev [unoptimized + debuginfo] target(s) in 0.72s
     Running target/debug/deps/wordutils-e01fe756921f1114

running 1 test
test tests::bench_initials ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/integration_tests-543eeef678725b84

running 1 test
test do_initials ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests wordutils

running 2 tests
test src/lib.rs - initials (line 11) ... ok
test src/lib.rs - initials (line 18) ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

To be completely honest, as a Rust neophyte, I am not entirely sure how beneficial it will be to have integration test support built-in like this. But I do like the idea that I can look in a crate's tests/ directory and get an idea of how the public API is supposed to work.

Conclusion

In this fifth post of the series, we have looked at three additional types of testing in Rust:

  • Benchmarking
  • Examples in documentation
  • Integration tests

For the most part, Rust's testing strategy is not much different than Go's. We don't see drastically different paradigms or conventions. But what we do see is perhaps a greater "ergonomic" in Rust, where examples are inside of the documentation instead of in the unit tests, and where integration tests are separated from unit tests, and designed to mirror the user experience of developers who use your libraries.