Oakwinter.com :: Code

Version 06. of the functional package has been pushed out to PyPI and the project’s own website. This release features:

• functional.flip will now reverse all non-keyword arguments, as opposed to simply reversing the first two as it did in version 0.5.

• functional.compose now comes with an optional unpack parameter to make up for some of the differences between Python and Haskell (which inspired most of the functional package), namely that Haskell functions are fully curried and Python functions usually aren’t.

  The unpack parameter means that you can now do something like this with compose:

  f(*g(*arg,**kw))

  i.e., automatically unpacking g’s return value and passing the result to f.

  To get this functionality, you’d write something like

  compose(f, g, unpack=True)(*args, **kwargs)

• Add weakref support to flip and compose objects in the C version.

• Sundry performance improvements for the C implementation.

In order to let people play with the latest changes to typecheck as soon as they’re available, some new goodies have been added to the site:

1. The development version of the website is now available for perusal. Why would you want this? Because the development version contains the docs for all the spiffy new features that will be coming out in the next release.

   The devel version of the site (yes, the whole site) can be found at http://oakwinter.com/code/typecheck/dev/.

2. In addition, the latest SVN revision (for trunk/) is now available in handy tarball form. The URL for the tarball will stay consistent, even when the tarball’s contents change. To figure out which revision you have, check the included REVISION file.

The latest change that you might want to play with is the new assert_type function. This is intended for in-code type checks, which can be a good deal more efficient than the traditional decorator-based checks. See the function’s docs for more details and examples.

While playing around with some metaprogramming stuff for the next version of typecheck, I discovered this:

>>> def foo(a=5):
...     return a
...
>>> foo()
5
>>> foo.func_defaults = (6,)
>>> foo()
6

This isn’t limited to change a keyword parameter’s default value, either:

>>> def foo(a, b):
...     return a, b
...
>>> foo.func_defaults = (4, 5)
>>> foo()
(4, 5)

Similarly, you can use this to turn keyword parameters into positional parameters:

>>> def foo(a, b=5):
...     return a, b
...
>>> foo.func_defaults = ()
>>> foo()
Traceback (most recent call last):
...
TypeError: foo() takes exactly 2 arguments (0 given)

Alright, this isn’t so much a “trick”, given that func_defaults is listed as writeable in the docs, but it’s still neat.

I’m having breakfast at a semi-regular coffee shop in Nashville, working on some Python bytecode-related black magic when the thought hits me: I want to tell someone about this, but who?

Given my experience in psychology research, my first instinct would be to write up a very scholarly paper and send it off for hopeful publication in a peer-reviewed journal or something. I somehow doubt, however, that any decent computer science journal would care about my Python ninja bytecode skills. What I needed, it seemed, was a computer science journal dedicated to Python.

The perl community already has one of these, appropriately titled The Perl Review. Published every three months, it features high-quality articles that show some serious perl-foo. As I understand it, these articles are generally written specifically for TPR and have to undergo both a stylistic and technical editorial process.

I’d like to see something like this happen for the Python world. I want some way of reading about cool stuff being done with and to Python, of finding articles with real technical merit. The status quo of Python journalism is blogs, with blog aggregators like Planet Python and Daily Python-URL being used to collect the mass of disparate postings into one place. I want some way of filtering the signal from the background static of the Python blogosphere.

In my mind, this “Python Quarterly” (name to be determined) would function on much the same lines as The Perl Review. There would be an editorial gauntlet to run, submissions would be expected to follow good writing standards, the whole shebang. Articles could cover any facet of Python development; some examples:

• In-depth looks at some facet of the CPython/PyPy/Jython/IronPython internals.

• Reports on experimental extensions to Python (e.g., Stackless Python).

• Python development best practices.

• Interviews with the shakers and movers of the Python community.

• Guided tours of new or existing modules/packages. This would include not just how to use the thing, but also its motivation and lessons learned during development.

• An op-ed page. This could be used (for example) to summarise any recent massive debates over the language’s future. python-dev, python-3000 and other mailing lists might be good raw sources for these.

Where would articles come from? Naturally, one source would be submissions that were crafted deliberately for the Python Quarterly. Another might be for the PQ’s editors to scan the various Python blog aggregators; upon noticing a quality post, they might invite the author to polish it up for inclusion in PQ’s next issue. Op-ed articles could be invited from the most vocal participants in recent python-{dev,3000,list} debates.

Other thoughts?

Not travelling around western Europe has done wonders for Cypress’s TODO list.

Most of my work lately has been going into designing (and redesigning) and implementing the Controller-side watch_module system. Put simply, this subsystem revolves around solving the first problem I outlined in the project introduction: how to ensure things like cache purity in the face of auto-reloaded modules.

This problem took a good while to solve, mostly because of ideological issues. To get into that, we first need a little background:

Cypress was originally conceived as a framework that would host other, higher-level applications. The goal was that these applications could run on top of Cypress without any modification whatsoever, totally ignorant of the fact that Cypress was running below them. Cypress would act as a mediator, sitting between the hosted application and the Python interpreter, intercepting module imports.

This import-interception was how Cypress was going to solve the three issues I mentioned in the project introduction. Cypress would redirect import requests to a specified revision control system (issue #2), periodically querying to find out which modules had changed since the last query and reloading the applicable chunks of code (issue #1); by having each application be hosted by its own instance of some Controller class, any number of applications could run on top of the same Cypress libraries (issue #3).

The part that was getting in the way was the goal of letting applications be ignorant of Cypress. I ran into this problem: say module M caches instances of a class C from module N. What happens when N is modified, changing the definition of class C? Module M’s cache now contains instances of both the old and new version of class C, leading to cache poisoning.

The solution that I’m going forward with tarnishes some of that ideological purity in exchange for more application-side power. If an application needs to know when a given module changes, it can call the watch_module method to tell Cypress, “call this object when anything happens to module foo.bar“. Given a module name and a callable, applications can now be signalled when the named module is edited, renamed or even deleted, giving the hosted code a chance to clear object caches, etc.

A more comprehensive design document is available from Cypress’s SVN repository.

Something I discovered/remembered the other day: when you define a Python function with keyword parameters, the default values are only created once, at function compilation-time. Now, since most default values are things like None or integers, this isn’t really a big deal. On the other hand, if the default value you’re supplying is MyClass(5, 6, 7), you might be surprised to discover that the same instance gets used with every function call.

We can exploit this to fake an equivalent to C’s static local variables. Try this:

def foo(item, acc=[]):
    acc.append(item)
    return acc

Now, call foo a few times with only a single argument.

>>> foo(5)
[5]
>>> foo(6)
[5, 6]
>>> foo(7)
[5, 6, 7]
>>>

I’m sure there are much more interesting uses for this than my contrived little example, but you get the idea.

That long-neglected ~/src/functional directory has finally guilted me back into action. Responding to a number of user suggestions, here are some things that will be going into the next release of functional:

• functional.flip will now reverse all non-keyword arguments, as opposed to simply reversing the first two as it does now.

• I’m going to provide some way of augmenting functional.compose to make up for some of the differences between Python and Haskell (which inspired most of the functional package), namely that Haskell functions are fully curried and Python functions usually aren’t.

  This difference means that you currently can’t do something like this with compose:

  f(*g(*arg,**kw))

  i.e., automatically unpacking g’s return value and passing the result to f.

  One idea I’m kicking around is an unpack keyword argument to compose. Setting this to a true value would result in g’s return value being unpacked before going to f, while setting it to false (the default) would pass the return value straight through. So, to get the functionality outlined above, you’d write something like

  compose(f, g, unpack=True)(*args, **kwargs)

• Add weakref support to flip and compose objects in the C version.

• Sundry performance improvements for the C implementation.

I’m hoping to have 0.6 out sometime early next week (maybe 10 or 11 April).

Why do a minor release (from 0.5 to 0.6) instead of a point release (from 0.5 to 0.5.1)? The changes to flip aren’t backwards-compatible, and I want to be sure people know that. Hopefully the current interface will settle down some, and things will tip toward doing more point releases than minor releases.

I’d like to give a proud-father-ly salute to Maxime Bourget’s typecheck frontend. Combining typecheck with the TPG parser generator, the frontend allows you to specify a function’s types using Haskell-like syntax in the callable’s docstring. To wit:

>>> from ize.typechecker import *
>>> @tize
... def f(x):
...     ':: int -> int'
...     return x+10
...
>>>

…which defines the function f as taking an int and returning an int.

All praise and suggestions go to Maxime Bourget (maxime dot bourget at gmail period com).

Despite not getting nearly the blog-post luvin’ that some of my other projects get, Cypress is one of the longest-running and most interesting pieces of code I work on. Time to spread the love around.

Cypress: not just another Python web framework.

In fact, despite being written to host a content management system for Community2.com, Cypress isn’t a web framework at all. Cypress is a new kind of framework, one targeted at three specific problems we encountered while developing C2:

• Having to restart Apache sucks. Whenever we change one of the static modules that power C2, we have to restart Apache. Now, I’ve heard that mod_perl is supposed to detect when your modules change and reload them automatically, but I’ve never been able to get it working, nor has anyone I know, and even if we could get it working, we wouldn’t use it.

  Why not just let mod_perl reload the modules? Because mod_perl reloads only the modules that have changed, not those that, say, cache instances of classes defined in the changed modules. This leads to cache poisoning, where your object cache has instances of both the new and old versions of the class. Cypress fixes this.

• Lack of real revision control sucks. Storing code in a database (the way the Everything Engine does) means you lose all the power that comes with a real revision control system. While Everything tries oh-so-desperately to make up for this with a half-assed versioning system, it just can’t compete with the likes of darcs or SVK.

  Cypress moves code out of the database and back under strong revision control, back where it belongs. Cypress is able to pull modules directly from revision control, and thanks to Cypress’ highly flexible architecture, you can have it work with any RCS you can imagine.

• One of the big problems we encountered with C2 is that there were other sites on the same server that used the Everything Engine. Because they all used the same set of modules, this meant that we couldn’t go changing things willy-nilly to suit C2. Cypress, again, fixes this.

I’m hoping to have Cypress ready sometime this decade, which seems to be a pretty slip-proof deadline.

While there’s no official website for Cypress, interested viewers can use the WebSVN viewer to watch the project’s progress.