A long time ago, in a blog post a few pages back in the archives, I spent a few paragraphs bemoaning Python’s unittest module and how it can’t be readily extended, nor can its extensions be easily composed. I gave as examples an extension that allows you to mark tests as “todo” and an extension that did reference counting around each test case (for C modules). While writing the extensions themselves was a little harder than I would have liked, the biggest problem was composing them — using both at the same time. Specifically, you can’t compose them, not without writing all-new code to merge the two functionalities. Consider:

TODO support:
        140 lines (5 core classes, 4 support classes/funcs)

Refcounting support:
        117 lines (4 core classes)

Composition:
        197 lines (6 core classes, 4 support classes/funcs)
        105 lines (3 classes of entirely new/rewritten code)

(All code snippets can be found in this directory. Code related to the old unittest design is in the before/ subdir, that related to the new design is in after/.)

test_harness, my new unittest package, was designed with flexibility and extensibility in mind. Using the same todo/refcounting examples from above:

TODO support:
        61 lines (1 core class, 4 support classes/funcs)

Refcounting support:
        36 lines (1 core class)

Composition:
        5 lines (1 core class, 3 imports)

That’s right: todo and refcounting support, with results written to stdout in five lines. And one of those lines is blank.

Where the new design really shines is in output. Unlike the old design — where you’d have to rewrite everything — changing your logging scheme from to-console to XML means changing this

    from test_harness import TextRunner
    from refcounting import RefcountRunner
    from todo import TodoRunner, TODO

    class OurRunner(TextRunner, RefcountRunner, TodoRunner):
        pass

to this:

    from xmlrunner import XmlTestRunner
    from refcounting import RefcountRunner
    from todo import TodoRunner, TODO

    class OurRunner(XmlTestRunner, RefcountRunner, TodoRunner):
        pass

That’s a two line change. That would have required a complete rewrite with the old system. Want both XML and to-console logging? Stick with the old unittest design and you’re looking at yet another rewrite. test_harness allows you to do this:

from test_harness import TextRunner
from xmlrunner import XmlTestRunner
from refcounting import RefcountRunner
from todo import TodoRunner, TODO

class OurRunner(TextRunner, XmlTestRunner, RefcountRunner, TodoRunner):
    pass

The biggest problem with the old unittest design is that, in trying to separate out the various concerns, it left the different components interconnected. TestCase objects depend on TestResult objects having certain methods; TestLoaders depend on your test case classes subclassing TestCase; TestRunners control which TestResult is used; etc. test_harness does away with this menagerie in favor of a single class: TestRunner. TestRunner objects are responsible for test suite iteration, running each individual test, collecting and categorizing any exceptions, and summarizing the results of the test run. Test loading/discovery is orthogonal to this process and as such is left to other packages, though rudimentary solutions are provided with the new package.

The biggest gripes about unittest I heard while researching unittest’s problems is that you a) have to subclass TestCase, and b) use TestCase methods to indicate test success/failure. In test_harness, there is no requirement to subclass TestCase (nor is there a TestCase class to subclass). Also, the usage of TestCase methods to signal failure — a consequence of the old TestCase/TestResult linkage — has been replaced with a test_harness.assertion submodule that contains functions like ok(), are_equal(), etc. Mapping old spellings to new:

    self.failUnless()               < = >    ok()
    self.assertEqual()              < = >    are_equal()
    self.failIfEqual()              < = >    are_not_equal()
    self.failUnlessAlmostEqual()    < = >    are_almost_equal()
    self.assertRaises()             < = >    raises()

Anyone interested is encouraged to play around with the new design. Comments to collinw at gmail point com