Atoum: A Different Unit Testing Framework for PHP
For a long time, it has seemed that when it comes to unit testing in PHP, there is only one game in town: PHPUnit. And in all fairness, it's popularity is justified. It is a good framework, and can generally handle standard use cases.
But recently I found Atoum. Not being one to start small, I began using it on a large (soon to be Open Source) project that I was writing from scratch.
Was it love or infatuation?
Atoum has some great out-of-the-box features:
- Support for PHP namespaces.
- Each test can be run directly.
- The CLI output is clean and intuitive.
- I love the method chaining support. <!--break--> The last one was a huge winner for me, as I could rapidly write tests and know that behind the scenes Atoum was performing rigid checking.
The learning curve for writing basic tests was tiny (keep in mind, I've written unit tests before in many languages using many frameworks -- YMMV). I had my first tests executing within minutes of beginning. And these weren't "hello world" tests, but legitimate tests against soon-to-be-production-grade code.
Losing the race
There were a few things I didn't particularly care for about Atoum. Some of the coding conventions conflicted with my own; the namespacing was a little rigid; error messages routinely contained less-than-stellar English. But I was willing to look past these.
In fact, I contributed a handful of patches back to the project, and the developers seem to be very interested in fostering community. I got the distinct feeling that they welcomed and would continue to welcome feedback.
But twenty or thirty tests in, small annoyances gave way to bigger annoyances. Code was not running properly. Tests were failing often, while my own (non-Atoum) scripts were running fine. I was spending far more time debugging and tinkering with Atoum than I was writing project code.
In several cases, the same suite of tests would fai on one run, then pass when run immediately after, and then fail on a subsequent run in a different way. I spent time chasing down dozens of spurious errors, changing code only to find that when I re-ran the tests, the original problem was gone, and a new one had arisen.
Then it dawned on me. Maybe the problem was in the environment.
I confirmed that what looked like a race condition between tests was indeed a race condition -- my tests were competing for some of the same resources, and were not being run (strictly) serially. This made complex setup and teardown very, very difficult. One test would setup, and another would begin using that test's setup. One test's fixtures could be torn down by another test. And since I was working with database-like resources, I couldn't easily fix this.
I ended up rewriting the tests in PHPUnit just to make sure I wasn't doing something stupid. And sure enough, the same tests that failed with Atoum passed with PHPUnit.
The moral of the story....
In the end, I converted all of my tests back to PHPUnit. I've lost some of the flexibility and elegance that I loved about Atoum, but writing tests once again became testing my code, not debugging my tests.
Atoum is a very promising library, and I like it a lot. With developers eager for patches, I know that I could have tracked down and fixed the problems, and see those fixes go right back into the project.
But there's the rub: I couldn't realistically do that. My project's timeline is not flexible enough to accomodate such a detour, and I can't, in good conscience, devote time to fixing one library when I know that there is another perfectly acceptible solution readily available.
I hope that in several months I will take a second look at Atoum and see that many of these problems have been fixed. But for now, my inner pragmatist has goaded me back into the fold of "stable and proven".
