Oakwinter.com :: Code

I’ve started working on a functional programming cookbook to illustrate practical uses for my functional package. So far I’ve got three examples up, with more on the way:

• Since functional’s shipped version of compose() only allows you to compose two functions, the cookbook includes a version of compose() that accepts multiple functions.

• I tend to use a heavily-functional style when writing __hash__() implementations, so I’ve included an example of how to do that using functional.

• Last but not least is a join() function that works like Python’s "".join(...) idiom, except that it automatically stringifies its arguments.

I’ve got some more examples in the works, including an explanation of when you should use the built-in reduce() over foldl() and vice-versa. Keep watching the cookbook for these and future Python functional programming goodies.

functional 0.7.0 has been released to PyPI and to the project’s own website. Included in this release:

• Several reference counting-related bugs were squashed.

• The C implementation now supports Python 2.3, in addition to 2.4 and 2.5.

• The test suite has been enhanced with fine-grained checking for reference counting bugs to try to keep the C version correct.

• Two new functions were added, map() and filter(), which function much like the Python built-ins of the same name — but without some of the built-ins’ weird behaviour.

Lastly, I know I said before that functional would include a functional.bwcompat submodule that contained exact copies of some Python builtins, the aim being to ease the transition to Python 3.0 for larger projects (when map() and filter() will no longer ship). I changed my mind for two reasons: 1) converting map() and filter() calls to list comprehensions will be part of a larger Python 2.x -> Python 3.x conversion, and 2) the more I waded in, the more I realised that Python’s built-ins have some really twisted semantics. So, instead of duplicating the builtins, I opted to write sane versions of these functions, which can be found at functional.map() and functional.filter().

Python Eggs of both versions, for Python 2.3 - 2.5, as well as tarballs, are available from PyPI.

As I mentioned in the last post, I’ve recently added reduce(), map() and filter() to functional. Since these mimic the semantics of the original Python built-ins, and since I wanted a sane version of map() at the same time, I decided to move reduce() and co. into a separate backwards compatibility-focused submodule, functional.bwcompat.

Now, this wasn’t a problem for the pure Python version of functional — you just add a new bwcompat.py in the functional/ directory. However, the C version turned out to be a bit trickier.

Most extension modules are just that — modules — meaning that there’s a foomodule.c file that creates an importable foo module. There’s little, if any, intentional support for creating packages at the C level. The popular way, as some googling turned up, is to create the root module, then manually add the submodules using PyModule_AddObject(). This works, but it means that all your submodules are imported as soon as you import the root namespace. It’s not a big deal, but it’s not ideal, either (for more background, see this python-dev thread.).

If you find yourself in the same situation, initfunctional() in functionalmodule.c should make a pretty good guide. A more stripped-down version can be found in this python-list post from Alex Martelli

.

I’ve recently set out to check off a number of the “remove x, y and z” items from PEP 3100, which details “miscellaneous Python 3.0 plans”. My latest of these was removing reduce(), map() and filter() from the built-in namespace.

The biggest hurdle wasn’t removing them a the C level; that took about 45 seconds. The main issue — which ended up taking 4 or 5 hours — was rewriting every single usage of map(), filter() and reduce() in the standard library. map() and filter() were pretty easy: you just change them to list comprehensions. reduce(), on the other hand, was a something else.

See, reduce() isn’t used that often in the stdlib, but when it is, it’s generally used in some hideous combination with map() and filter() that makes it extremely time consuming to rewrite. What makes this even better is that these chunks of code are usually in places without test coverage, meaning there’s no easy way to verify that what I’ve done is right.

To make life easier for large projects that make heavy use of map() and friends, I’m adding a functional.bwcompat submodule to functional. This will contain map(), reduce(), filter() and any other functional programming-related tools that will get kicked out of mainline Python come the 3.0 release. All you’ll need to do to make your code work is a simple

from functional.bwcompat import map, reduce, filter

and everything will work as before.

functional.bwcompat will make its debut in functional’s 0.7 release.

Version 06. of the functional package has been pushed out to PyPI and the project’s own website. This release features:

• functional.flip will now reverse all non-keyword arguments, as opposed to simply reversing the first two as it did in version 0.5.

• functional.compose now comes with an optional unpack parameter to make up for some of the differences between Python and Haskell (which inspired most of the functional package), namely that Haskell functions are fully curried and Python functions usually aren’t.

  The unpack parameter means that you can now do something like this with compose:

  f(*g(*arg,**kw))

  i.e., automatically unpacking g’s return value and passing the result to f.

  To get this functionality, you’d write something like

  compose(f, g, unpack=True)(*args, **kwargs)

• Add weakref support to flip and compose objects in the C version.

• Sundry performance improvements for the C implementation.

That long-neglected ~/src/functional directory has finally guilted me back into action. Responding to a number of user suggestions, here are some things that will be going into the next release of functional:

• functional.flip will now reverse all non-keyword arguments, as opposed to simply reversing the first two as it does now.

• I’m going to provide some way of augmenting functional.compose to make up for some of the differences between Python and Haskell (which inspired most of the functional package), namely that Haskell functions are fully curried and Python functions usually aren’t.

  This difference means that you currently can’t do something like this with compose:

  f(*g(*arg,**kw))

  i.e., automatically unpacking g’s return value and passing the result to f.

  One idea I’m kicking around is an unpack keyword argument to compose. Setting this to a true value would result in g’s return value being unpacked before going to f, while setting it to false (the default) would pass the return value straight through. So, to get the functionality outlined above, you’d write something like

  compose(f, g, unpack=True)(*args, **kwargs)

• Add weakref support to flip and compose objects in the C version.

• Sundry performance improvements for the C implementation.

I’m hoping to have 0.6 out sometime early next week (maybe 10 or 11 April).

Why do a minor release (from 0.5 to 0.6) instead of a point release (from 0.5 to 0.5.1)? The changes to flip aren’t backwards-compatible, and I want to be sure people know that. Hopefully the current interface will settle down some, and things will tip toward doing more point releases than minor releases.

The functional project has seen a lot of work since its inception (which predates its announcement). Version 0.1 was released on 19 January, and now less than two weeks later, version 0.5 is making its way to PyPI.

This latest release sees the project split into two branches. One is written, as before, in pure Python, targeting readability and portability. The second is coded in C, aiming for raw performance. Both support the same functionality and pass the same test suite.

Speaking of functionality…I was digging around the itertools module the other day and what should I discover but pre-existing implementations of most of functional’s members. All the duplicates have been stripped out for the 0.5 release. I’m hoping to add more useful higher-order tools to replace them.

Since I clearly don’t have enough on my plate, I’ve started a new project: functional, an effort to bring functional programming tools to Python.

functional provides a pure-Python implementation of numerous tools common in functional programming, such as foldl, foldr, take, flip, as well as mechanisms for partial function application and function composition.

In addition, this project serves as a test-bed for the functional module that will be shipped with Python 2.5. While the module to be shipped with Python will be partially written in C for speed, my module is written in Python to gain readability and portability.

While the functional project is currently pure Python, future releases will see the project broken into two branches. One will stay pure Python, aiming for maximum portability and readability, while the other will be a C/Python hybrid designed for speed and real-world deployment. Both will offer identical functionality, differing only in their implementation language.

This project began as a single patch to the C-language module in core Python that added strict implementations of foldl and foldr. I quickly replaced that patch with one that added over a dozen more functions and classes to the module. That patch, unfortunately, is still sitting in Python’s patch queue. While I’m still hopeful that my work will make it into mainline Python, I’ll be tending this project in the meantime, allowing people to start using these tools even before Python 2.5 comes out.