Oakwinter.com :: Code

So, there was originally supposed to be a Perl version of svnmock included in this release. I spent a day or two working on it before remembering how much I hate Perl. svk rm branches/perl felt somehow purifying.

On the upside, recoding the package from scratch gave me some good ideas about how to improve some things in the Python version. The main change in the 0.3 release is that you no longer have to explicity declare and directly manipulate a MockSession object; svnmock.mock takes care of creating the inital MockSession for you, then provides convenience add_command, add_error, etc, module-level functions to manipulate it. This turns out to be a big win in practice.

In other good news, svnmock’s internals will be serving as the model for the rewrite of DBD::Mock that I’m working on. DBD::Mock’s guts are currently…well…not something you’d bring home for dinner at your parents’. Again, this will be a big win, both in terms of code maintainability and in functionality.

Well, released a few days ago, but still.

The big news for 0.2 is the addition of tracing support and a rearrangement of the internals to support it. svnmock.trace allows you to keep a running log of which API functions have been called, with what arguments and what it returned (it logs raised exceptions, too!). You can then pretty-print the log and examine it to make sure things are going as planned.

Why would you want to do this? It’s great for quick debugging work when you don’t want to spend quality time with svnmock.mock, scripting up an entire session. That session may involve thousands of API calls, whereas svnmock.trace can skip directly to the area you’re concerned about.

On the horizon for 0.3:

• A tool to turn the output captured by svnmock.trace into a MockSession object.

• A perl port. svnmock -> SVN::Mock.

After spending this past week madly writing documentation for svnmock, I’ve finally finished the last bit of the reference manual. Version 0.1, the initial public release, is currently making its way to SourceForge, the Python Package Index and the project’s own site.

Version 0.2 should be out fairly soon (if not later today). It will include support for tracing API calls, as opposed to making assertions about them.

As part of the push to get an initial release of svnmock out the door, I’ve spent the last two weeks or so working on a website for the project. The biggest part of this effort has been writing documentation, particularly a tutorial, to explain this silly package.

By and large, the documentation has been fairly easy to write. I’m pretty good at, and enjoy, the very precise, technical style needed to write exacting specifications. I also look at spec writing as an opportunity to double-check the test suite; for every claim about how a given piece of code operates, I add a footnote pointing to the area of the test suite that verifies that claim. True, this makes things take longer than they otherwise would, but I’ve found several untested bits of code this way.

The tutorial section, on the other hand, has been torture. Whereas spec writing involves merely throwing down everything I know about the code in the most lawyer-like language I can summon, crafting a tutorial requires writing about the project as if I knew nothing about it and were exploring it for the first time. It can get frustrating, trying to distill an entire software package into simple, little words.

A similar effort has been underway for typecheck, as well. I’m pushing to have the project ready for publication on comp.lang.python by 0.4, our next minor-version release. We’ve recently brought David Wheeler on board, and he’s given me some good insights as to things we can do better.

One of the big problems he’s pointed out so far is the package’s documentation. It sucks, and you’ll get no argument from me about that. The docs have long consisted of a single, pages-long document that skims the package’s functionality. This format is left over from Iain Lowe’s initial effort, and while it may have worked in the past, the current version of the package is too large, too wide-ranging, to fit in a single page.

My current effort has involved breaking the current document into a sectioned-off tutorial and reference manual. As a result, it’s much better organised now, with a hint of “flow” and a dash of “coherence”. Also, I’m spending a lot of time working on more realisitic examples; as it is, a lot of the demos use the int and float types, which makes for simple examples, but they’re not very realistic.

I’m hoping to get the initial draft out the door some time today, then open the floor for comments and revisions. Mmm…open source.

So, the other day, I was minding my own business, just writing some test cases for Cypress’s new storage.svn.local module, as I do. I kept getting strange errors from svnmock that a certain API function isn’t being called with the right arguments. After a while, I eventually track down the problem to a chunk of code iterating over a dictionary, meaning that the iteration isn’t guaranteed to be stable.

This was something that I hadn’t anticipated when designing the API, the need to say “the following commands need to be executed, but I don’t care in what order”. The next few days are a blur, resulting in a near-total reworking of svnmock’s internals. The end result looks very much like typecheck’s internals: a set of plug-in style classes that do all the work, controlled by a single dispatcher.

One of these classes is AnyOrder, which is designed to solve the “run the following commands in whatever order you can” problem. This class proved tricky to implement, primarily because I was underestimating what kind of mechanism it would require. My initial test cases were too easy; the machinery needed to solve:

sequence_1 = Seq([command_1, command_2])
sequence_2 = Seq([command_3, command_4])

AnyOrder([sequence_1, sequence_2])

is simple, because the first commands in each sequence are different. On the other hand,

sequence_1 = Seq([command_1, command_2])
sequence_2 = Seq([command_1, command_3])

AnyOrder([sequence_1, sequence_2])

is significantly harder, since the first commands are the same. In the end, I solved this with a backtracking-capable state machine. Actually coding it wasn’t the hard part, though: the hard part was realising I needed a backtracking state machine in the first place.

The most recent problem has been how to reconcile svnmock’s return values with the AnyOrder mechanism. Recall that you can do something like

pool = session.add(core.svn_pool_create, [None])
scratch_pool = session.add(core.svn_pool_create, [pool])

and svnmock will assert that the scratch pool is indeed created out of the first pool (by tracking the return value from the first core.apr_pool_create() call). It turns out that this becomes tricky when you introduce backtracking in to the mix (answering “why” requires a whiteboard to illustrate).

The current solution to this involves replacing the old return value scheme with a dedicated Return class. Instances of the class store information about the function that returned them, such as the exact API function and its arguments. This works in most cases, but falls apart in situations like this:

pool_1 = core.svn_pool_create(None)
pool_2 = core.svn_pool_create(None)
scratch = core.svn_pool_create(pool_1)

Since both pool_1 and pool_2 are created with the same API function and arguments, there’s no way to assert that scratch is generated from pool_1 as opposed to pool_2. I’m not sure if this is common enough to worry about, though; I suppose if this little module catches on, and this particular bug starts causing people headaches, I’ll be more motivated to fix it.

With the new design of svnmock stabilised, I spent some time yesterday putting it into service, testing a new Subversion-targeting backend for Cypress. There’ve been some minor refinements (e.g., making the args parameter to MockSession.add optional), and here are my observations thus far:

• The low-level nature of svnmock is both a blessing and a curse. The blessing: exact, super-precise assertions as to which API functions should be called. The curse: you effectively end up duplicating the code you’re testing. I’m considering adding support for something like, “I don’t care what the next four API calls are, but the fifth one needs to be X(Y, Z)”, but I’m not sure that’s a road I want to head down. On the other hand…

• …it’s gratifyingly easy to build macro-methods on top of these low-level primitives, and my test cases have started sprouting private methods to manipulate the appropriate MockSession objects. Rather than endlessly duplicate the MockSession.add() calls needed to test method X, those calls get moved into a mock_X() method on the test case. While this certainly cleans things up, I’m looking for a way to shift more of this burden to the svnmock.mock module.

In the process of testing Cypress, I discovered bugs in Python, Subversion’s python API and SWIG, the tool used to automatically generate Subversion’s bindings for python, perl and ruby. The Subversion bug: there’s a function in the Subversion python bindings with an illegal name, svn.client.import (”import” is a keyword in python, meaning it’s illegal to use it as an identifier). The python bug: the internal function used to register C-language extension modules, Py_InitModule, doesn’t check to make sure that functions/methods/whatevers use legal names. The SWIG bug: SWIG fails to warn adequately when you try to generate an illegally-named function.

I first reported the bug to the python-dev mailing list, offering to patch the bug it myself if there was interest in a fix. The word I got there was that, given this problem’s extreme rarity, it wasn’t worth the time (which I agree with). Next, I sent an email to the general Subversion mailing list, pointing out that this function needed to be renamed; the only response I’ve received to date was that the new name I proposed wouldn’t work, but nothing more constructive. Finally, I posted a bug report to the SWIG project’s bug tracker; the developer in charge of the python backend commented quickly on the bug report, saying that it’s a well-established behaviour of SWIG to allow you to create illegal names, though it does warn about it.

Not exactly the response I was hoping for. Sigh.

While on the train back from Frankfurt this morning, I realised that I had been designing the svnmock API all wrong. The old design was based on simply populating a mock repository in RAM, then letting the SVN-emulation layer interact with that repository. I hadn’t gotten very far into implementing the design, but already I could tell that it was going to be a massive, unwieldly operation. Worse yet, I realised that it wouldn’t offer the level of control I myself would want as a tester. The rest of the trip was spent looking at snow-covered Hessian woods and restarting the API design process from scratch.

While talking over possible design issues with a guy I graduated from uni with, Tyler Hall, I had a brainstorm. Within about 15 minutes, I had ripped out the entire old design, several hundred lines of complicated class interaction (which was only a tiny fraction of what would be needed), and had replaced it with totally new workings. Result: in less than 100 lines, I had a mock up of the entire Subversion API. Better yet, it would be totally future-proof: if the Subversion python API changes radically, I don’t have to release a new version — the existing code will adapt on its own.

Where the old design relied on me knowing what every single function in the API does, the new design is assumes that you know what all these silly procedures do. It works like so:

1. You create a new MockSession object.

2. You tell the Session object that you expect function X to be run with parameters Y and Z, and that it ought to return 7; anything else is to be considered an error, and it should blow up.

An example:

from svnmock import mock, core, repos, fs

ses = mock.MockSession()

pool = ses.add(core.svn_pool_create, [None])
scratch_pool = ses.add(core.svn_pool_create, [pool])

We’ll take this slow:

1. Line 1: svnmock.mock is the module used to populate the testing environment. svnmock.core, svnmock.repos and svnmock.fs are modules holding the emulated API functions and constants. We need to import them so we can refer to their functions later.

2. Line 3: create a new MockSession object. MockSession instances are used to organise our test environments; having them be objects — as opposed to a more imperative-style interface — allows us to easily swap test environments in and out, making them reusable.

3. Lines 5-6: here’s where things start to get interesting. In line 5, we specify that the first command to be executed must be core.svn_pool_create, with the sole parameter of None. Line 6 specifies that core.svn_pool_create will be run again, but this time the parameter will — nay, must — be the return value from the first svn_pool_create() call; anything else is to be treated as an error.

See the doc/mock_session.py file in the SVN repository for further examples of the new syntax.

One of the problems with writing code that interacts with a database (for example) is that it’s hard to test; do you have your test suite create, populate and destroy whole databases to test your code against? That’s an awful lot of work. Then there’s the question about testing those tricky edge cases, like certain error conditions: if you’re creating new databases, how do you simulate, say, your database server suffering a catastrophic failure. I can think of one solution, but it would get pretty expensive pretty quickly.

Fortunately, since this is a common enough problem (testing database interaction), smart people have already solved it. In the perl community, for example, DBD::Mock (full disclosure: I work on DBD::Mock some) is one fairly well-known way of testing interaction with database systems. It allows you to say, “Make sure the following SQL statements are executed in the following order, and return these results”. It can also be used to simulate what happens when the connection to the database dies mid-query, for example.

Once you move away from databases, though, to other, though still complex systems, the range of testing options becomes more limited. In my case, I wanted to write several applications and support libraries on top of the Python bindings for Subversion, a popular revision control system. When it came time to write tests for the Subversion-facing code, I faced a dilemma: was I back to creating full-blown SVN repositories for each test case?

My solution was svnmock. The svnmock package does for Subversion’s Python API what DBD::Mock does for perl’s DBI: it makes testing easy. svnmock allows the test writer to say, like with DBD::Mock, “I want the following series of function calls, with this set of parameters and this return value”. While this may seem fairly low-level, it is trivial to write macro-like constructs on top of the current set of primitives. One of my favourite features of svnmock is that it allows you to specify that the return value from api_func_1() must be used as a parameter to api_func_2().

svnmock’s project website may be found here.