Oakwinter.com :: Code

Since I clearly don’t have enough on my plate, I’ve started a new project: functional, an effort to bring functional programming tools to Python.

functional provides a pure-Python implementation of numerous tools common in functional programming, such as foldl, foldr, take, flip, as well as mechanisms for partial function application and function composition.

In addition, this project serves as a test-bed for the functional module that will be shipped with Python 2.5. While the module to be shipped with Python will be partially written in C for speed, my module is written in Python to gain readability and portability.

While the functional project is currently pure Python, future releases will see the project broken into two branches. One will stay pure Python, aiming for maximum portability and readability, while the other will be a C/Python hybrid designed for speed and real-world deployment. Both will offer identical functionality, differing only in their implementation language.

This project began as a single patch to the C-language module in core Python that added strict implementations of foldl and foldr. I quickly replaced that patch with one that added over a dozen more functions and classes to the module. That patch, unfortunately, is still sitting in Python’s patch queue. While I’m still hopeful that my work will make it into mainline Python, I’ll be tending this project in the meantime, allowing people to start using these tools even before Python 2.5 comes out.

After spending this past week madly writing documentation for svnmock, I’ve finally finished the last bit of the reference manual. Version 0.1, the initial public release, is currently making its way to SourceForge, the Python Package Index and the project’s own site.

Version 0.2 should be out fairly soon (if not later today). It will include support for tracing API calls, as opposed to making assertions about them.

First, the typecheck project — both the website and Subversion repository — have been moved to oakwinter.com. The new URL is http://oakwinter.com/code/typecheck.

Second, the website has been given a thorough scrubbing. The documentation has been greatly cleaned up and has been broken into separate tutorial and reference manual sections.

Last but not least, typecheck release 0.3.4 has been shipped off to SourceForge, PyPI and the project’s own website. This new release features the concept of typeclasses, a way of grouping classes and types based on functionality. See the documenation for details.

This will probably be the last release in the 0.3.x series. Next up will be the 0.4 release; this version will shed a lot of antiquated backwards compatibility mechanisms (for values of “antiquated” approaching “version 0.1.6″) and a general streamlining of the package’s functionality.

As part of the push to get an initial release of svnmock out the door, I’ve spent the last two weeks or so working on a website for the project. The biggest part of this effort has been writing documentation, particularly a tutorial, to explain this silly package.

By and large, the documentation has been fairly easy to write. I’m pretty good at, and enjoy, the very precise, technical style needed to write exacting specifications. I also look at spec writing as an opportunity to double-check the test suite; for every claim about how a given piece of code operates, I add a footnote pointing to the area of the test suite that verifies that claim. True, this makes things take longer than they otherwise would, but I’ve found several untested bits of code this way.

The tutorial section, on the other hand, has been torture. Whereas spec writing involves merely throwing down everything I know about the code in the most lawyer-like language I can summon, crafting a tutorial requires writing about the project as if I knew nothing about it and were exploring it for the first time. It can get frustrating, trying to distill an entire software package into simple, little words.

A similar effort has been underway for typecheck, as well. I’m pushing to have the project ready for publication on comp.lang.python by 0.4, our next minor-version release. We’ve recently brought David Wheeler on board, and he’s given me some good insights as to things we can do better.

One of the big problems he’s pointed out so far is the package’s documentation. It sucks, and you’ll get no argument from me about that. The docs have long consisted of a single, pages-long document that skims the package’s functionality. This format is left over from Iain Lowe’s initial effort, and while it may have worked in the past, the current version of the package is too large, too wide-ranging, to fit in a single page.

My current effort has involved breaking the current document into a sectioned-off tutorial and reference manual. As a result, it’s much better organised now, with a hint of “flow” and a dash of “coherence”. Also, I’m spending a lot of time working on more realisitic examples; as it is, a lot of the demos use the int and float types, which makes for simple examples, but they’re not very realistic.

I’m hoping to get the initial draft out the door some time today, then open the floor for comments and revisions. Mmm…open source.

So, the other day, I was minding my own business, just writing some test cases for Cypress’s new storage.svn.local module, as I do. I kept getting strange errors from svnmock that a certain API function isn’t being called with the right arguments. After a while, I eventually track down the problem to a chunk of code iterating over a dictionary, meaning that the iteration isn’t guaranteed to be stable.

This was something that I hadn’t anticipated when designing the API, the need to say “the following commands need to be executed, but I don’t care in what order”. The next few days are a blur, resulting in a near-total reworking of svnmock’s internals. The end result looks very much like typecheck’s internals: a set of plug-in style classes that do all the work, controlled by a single dispatcher.

One of these classes is AnyOrder, which is designed to solve the “run the following commands in whatever order you can” problem. This class proved tricky to implement, primarily because I was underestimating what kind of mechanism it would require. My initial test cases were too easy; the machinery needed to solve:

sequence_1 = Seq([command_1, command_2])
sequence_2 = Seq([command_3, command_4])

AnyOrder([sequence_1, sequence_2])

is simple, because the first commands in each sequence are different. On the other hand,

sequence_1 = Seq([command_1, command_2])
sequence_2 = Seq([command_1, command_3])

AnyOrder([sequence_1, sequence_2])

is significantly harder, since the first commands are the same. In the end, I solved this with a backtracking-capable state machine. Actually coding it wasn’t the hard part, though: the hard part was realising I needed a backtracking state machine in the first place.

The most recent problem has been how to reconcile svnmock’s return values with the AnyOrder mechanism. Recall that you can do something like

pool = session.add(core.svn_pool_create, [None])
scratch_pool = session.add(core.svn_pool_create, [pool])

and svnmock will assert that the scratch pool is indeed created out of the first pool (by tracking the return value from the first core.apr_pool_create() call). It turns out that this becomes tricky when you introduce backtracking in to the mix (answering “why” requires a whiteboard to illustrate).

The current solution to this involves replacing the old return value scheme with a dedicated Return class. Instances of the class store information about the function that returned them, such as the exact API function and its arguments. This works in most cases, but falls apart in situations like this:

pool_1 = core.svn_pool_create(None)
pool_2 = core.svn_pool_create(None)
scratch = core.svn_pool_create(pool_1)

Since both pool_1 and pool_2 are created with the same API function and arguments, there’s no way to assert that scratch is generated from pool_1 as opposed to pool_2. I’m not sure if this is common enough to worry about, though; I suppose if this little module catches on, and this particular bug starts causing people headaches, I’ll be more motivated to fix it.