Oakwinter.com :: Code

With the new design of svnmock stabilised, I spent some time yesterday putting it into service, testing a new Subversion-targeting backend for Cypress. There’ve been some minor refinements (e.g., making the args parameter to MockSession.add optional), and here are my observations thus far:

• The low-level nature of svnmock is both a blessing and a curse. The blessing: exact, super-precise assertions as to which API functions should be called. The curse: you effectively end up duplicating the code you’re testing. I’m considering adding support for something like, “I don’t care what the next four API calls are, but the fifth one needs to be X(Y, Z)”, but I’m not sure that’s a road I want to head down. On the other hand…

• …it’s gratifyingly easy to build macro-methods on top of these low-level primitives, and my test cases have started sprouting private methods to manipulate the appropriate MockSession objects. Rather than endlessly duplicate the MockSession.add() calls needed to test method X, those calls get moved into a mock_X() method on the test case. While this certainly cleans things up, I’m looking for a way to shift more of this burden to the svnmock.mock module.

In the process of testing Cypress, I discovered bugs in Python, Subversion’s python API and SWIG, the tool used to automatically generate Subversion’s bindings for python, perl and ruby. The Subversion bug: there’s a function in the Subversion python bindings with an illegal name, svn.client.import (”import” is a keyword in python, meaning it’s illegal to use it as an identifier). The python bug: the internal function used to register C-language extension modules, Py_InitModule, doesn’t check to make sure that functions/methods/whatevers use legal names. The SWIG bug: SWIG fails to warn adequately when you try to generate an illegally-named function.

I first reported the bug to the python-dev mailing list, offering to patch the bug it myself if there was interest in a fix. The word I got there was that, given this problem’s extreme rarity, it wasn’t worth the time (which I agree with). Next, I sent an email to the general Subversion mailing list, pointing out that this function needed to be renamed; the only response I’ve received to date was that the new name I proposed wouldn’t work, but nothing more constructive. Finally, I posted a bug report to the SWIG project’s bug tracker; the developer in charge of the python backend commented quickly on the bug report, saying that it’s a well-established behaviour of SWIG to allow you to create illegal names, though it does warn about it.

Not exactly the response I was hoping for. Sigh.

While on the train back from Frankfurt this morning, I realised that I had been designing the svnmock API all wrong. The old design was based on simply populating a mock repository in RAM, then letting the SVN-emulation layer interact with that repository. I hadn’t gotten very far into implementing the design, but already I could tell that it was going to be a massive, unwieldly operation. Worse yet, I realised that it wouldn’t offer the level of control I myself would want as a tester. The rest of the trip was spent looking at snow-covered Hessian woods and restarting the API design process from scratch.

While talking over possible design issues with a guy I graduated from uni with, Tyler Hall, I had a brainstorm. Within about 15 minutes, I had ripped out the entire old design, several hundred lines of complicated class interaction (which was only a tiny fraction of what would be needed), and had replaced it with totally new workings. Result: in less than 100 lines, I had a mock up of the entire Subversion API. Better yet, it would be totally future-proof: if the Subversion python API changes radically, I don’t have to release a new version — the existing code will adapt on its own.

Where the old design relied on me knowing what every single function in the API does, the new design is assumes that you know what all these silly procedures do. It works like so:

1. You create a new MockSession object.

2. You tell the Session object that you expect function X to be run with parameters Y and Z, and that it ought to return 7; anything else is to be considered an error, and it should blow up.

An example:

from svnmock import mock, core, repos, fs

ses = mock.MockSession()

pool = ses.add(core.svn_pool_create, [None])
scratch_pool = ses.add(core.svn_pool_create, [pool])

We’ll take this slow:

1. Line 1: svnmock.mock is the module used to populate the testing environment. svnmock.core, svnmock.repos and svnmock.fs are modules holding the emulated API functions and constants. We need to import them so we can refer to their functions later.

2. Line 3: create a new MockSession object. MockSession instances are used to organise our test environments; having them be objects — as opposed to a more imperative-style interface — allows us to easily swap test environments in and out, making them reusable.

3. Lines 5-6: here’s where things start to get interesting. In line 5, we specify that the first command to be executed must be core.svn_pool_create, with the sole parameter of None. Line 6 specifies that core.svn_pool_create will be run again, but this time the parameter will — nay, must — be the return value from the first svn_pool_create() call; anything else is to be treated as an error.

See the doc/mock_session.py file in the SVN repository for further examples of the new syntax.

One of the problems with writing code that interacts with a database (for example) is that it’s hard to test; do you have your test suite create, populate and destroy whole databases to test your code against? That’s an awful lot of work. Then there’s the question about testing those tricky edge cases, like certain error conditions: if you’re creating new databases, how do you simulate, say, your database server suffering a catastrophic failure. I can think of one solution, but it would get pretty expensive pretty quickly.

Fortunately, since this is a common enough problem (testing database interaction), smart people have already solved it. In the perl community, for example, DBD::Mock (full disclosure: I work on DBD::Mock some) is one fairly well-known way of testing interaction with database systems. It allows you to say, “Make sure the following SQL statements are executed in the following order, and return these results”. It can also be used to simulate what happens when the connection to the database dies mid-query, for example.

Once you move away from databases, though, to other, though still complex systems, the range of testing options becomes more limited. In my case, I wanted to write several applications and support libraries on top of the Python bindings for Subversion, a popular revision control system. When it came time to write tests for the Subversion-facing code, I faced a dilemma: was I back to creating full-blown SVN repositories for each test case?

My solution was svnmock. The svnmock package does for Subversion’s Python API what DBD::Mock does for perl’s DBI: it makes testing easy. svnmock allows the test writer to say, like with DBD::Mock, “I want the following series of function calls, with this set of parameters and this return value”. While this may seem fairly low-level, it is trivial to write macro-like constructs on top of the current set of primitives. One of my favourite features of svnmock is that it allows you to specify that the return value from api_func_1() must be used as a parameter to api_func_2().

svnmock’s project website may be found here.

The typecheck project grew out of a desire to gain more safety in my python programs. Whereas the majority of the python community seems to be in love with (or at least, “ok with”) the idea of duck-typing, duck-typing has always struck me as a post-fact rationalisation of the decision to make python a weakly-typed language. In less fancy language: I hate duck-typing so much, I’m willing to spend a few months developing a way to get away from it.

So, what is the typecheck package? At its most primative, it provides a way to make sure the arguments to your functions, methods and generators (and their return values) are of the expected types. Going further, it offers a wealth of extensibility, allowing — and expecting — programmers to push it much further than simple “is argument ‘a’ an integer?” typechecking. Beyond this rudimentary mode of operation, the typecheck package ships with the following tools:

• Utility classes allowing the programmer to build up more complex “types” from boolean expressions. Though the And(), Not() and Or() classes can of course be used with all built-in and user-defined types, they can also be nested within one another to produce all other boolean operators.

• Other utility classes exist to enforce duck-typed constraints. For example, IsCallable() asserts that the object can be invoked like a function, and IsIterable() ensures that you’ll be able to utilise the object in for loops. A HasAttr() class is provided, allowing assertions that the object has certain, specified attributes.

• The typecheck package borrows the concept of “type variables” from languages like Haskell. Type variables allow you to say, “I don’t care what type parameter ‘a’ is, but parameters ‘c’ and ‘d’ need to be of that same type, as does the function’s return value.”

Beyond these standard tools, care has been taken to allow, for example, user-defined container classes to hook into the typechecker, providing type safety for the most exotic of classes. For simple iterator container classes, mixins exist to provide drop-in typechecking capabilities.

The project’s website is: http://oakwinter.com/code/typecheck/. There, you’ll find links to more complete documentation, the project’s Subversion archive, and source and Python Egg distributions.

My involvement - the original typecheck project was developed by Iain Lowe. In Fall 2005, I started working on my own typechecker to address certain limitations of Iain’s code, though also intending to keep full, interface-level backwards compatibility with what he had done. In November 2005, I felt my codebase was complete enough for a 0.1 release and so contacted Iain about merging our projects. He agreed, and my code became the core of the combined typechecker. I’m currently leading the coding and documentation work on the project.