function annotations


To anyone planning to email me about how much you hate the syntax for return value annotations: don’t. Guido wants the -> arrow, so the arrow is what we’re getting. Your ideas for using returns or return or whatever else have already occurred to others — namely me — and were rejected months ago.

Guido’s the one you have to convince, not me, and he’s already made up his mind.

Based on this python-3000 thread and a number of off-list emails, I’m dropping my earlier objection to PEP 3107. I hadn’t been convinced that there was a sufficiently broad spectrum of use-cases for function annotations to justify changing Python’s syntax. A bunch of people came out of the woodwork with viable uses for annotations, which is what I was looking for. Accordingly, I’ll be working up a patch to PEP 3107 to include a “Use Cases” section.

Thanks to everyone who emailed or commented, especially Phillip J. Eby, who led the python-3000 effort to convince me.

A blogified version of a python-3000 post, in which the author of PEP 3107 revels in situational irony.

I was explaining function annotations to a friend this past weekend and found that, even though I had written a PEP on the subject and spent months debating the little details of “how are we going to make annotations work?”, I was hard-pressed to answer the question of “why are we doing this?”

The biggest problem I faced — then and now — is justifying the use-cases for annotations. Here’re the use-cases I could come up with off the top of my head: information for typecheckers; doc strings for parameters; extra information for IDEs; extra information for static analysis tools like pylint. These can all be addressed together:

Are the users clamoring for these things? Do these address real problems that users are having?

Not to my knowledge.

  1. Information for typecheckers

    In a recent python-ideas post, Guido van Rossum said that “Collin’s existing type annotation library … could be made more elegant by attaching the types directly to the arguments”. As far as I can tell, the only gains in elegance are that you don’t have to repeat the names of a function’s parameters in the typechecking decorator. None of my users have ever complained about this tiny bit of repetition, and I’ve never felt it an undue burden in my own usage.

    It could even be considered an advantage, since including the “annotations” in the typechecking decorator means all I have to do to remove typechecking from a function is delete a single line, rather than pick through a function’s declaration, removing the relevant bits.

  2. Doc strings for parameters

    def foo(a: 'the object to be frobnicated',
                b=7: 'this controls the level of frobnication',
                c='Rojo': 'the name of a color, in Spanish, that should
    be applied to the 
                      frobnicated thing') -> 'Returns an integer between
    9 and 13':
       ''' Frobnicate an object in a Spanish way '''
       ...

    What does this accomplish that can’t be achieved with any of the standard documentation syntaxes in existence today?

  3. Type information for IDEs

    I can see it being genuinely useful to be able to get parameter/return type information in a tooltip message. But IDLE can do this already, without annotations:

  4. Type information for static analysis tools

    Quoting Nick Coghlan, from August 2006: “annotations wouldn’t be useful for tools like pychecker … to be really useful for a tool like pychecker they’d have to be ubiquitous, and that’s really not Python any more”. Agreed.

I could say You Aren’t Going to Need It, but that gets the tense wrong; we’re getting along without annotations quite nicely here in the present. In short: I think that PEP 3107 be rejected as an overly-specific, unnecessary addition to the language.

Anyone with thoughts on or responses to this article should post them to the python-3000 mailing list.

My very first PEP, fresh and piping-hot:



PEP: 3107
Title: Function Annotations
Version: 53169
Last-Modified: 2006-12-27 22:59:16 -0600 (Wed, 27 Dec 2006)
Author: Collin Winter <collinw at gmail.com>,
Tony Lownds <tony at lownds.com>
Status: Draft
Type: Standards Track
Requires: 362
Content-Type: text/x-rst
Created: 2-Dec-2006
Python-Version: 3.0
Post-History:

Abstract

This PEP introduces a syntax for adding arbitrary metadata annotations
to Python functions [1].

Rationale

Because Python’s 2.x series lacks a standard way of annotating a
function’s parameters and return values (e.g., with information about
what type a function’s return value should be), a variety of tools
and libraries have appeared to fill this gap [2]. Some
utilise the decorators introduced in "PEP 318", while others parse a
function’s docstring, looking for annotations there.

This PEP aims to provide a single, standard way of specifying this
information, reducing the confusion caused by the wide variation in
mechanism and syntax that has existed until this point.

Fundamentals of Function Annotations

Before launching into a discussion of the precise ins and outs of
Python 3.0’s function annotations, let’s first talk broadly about
what annotations are and are not:

  1. Function annotations, both for parameters and return values, are
    completely optional.

  2. Function annotations are nothing more than a way of associating
    arbitrary Python expressions with various parts of a function at
    compile-time.

    By itself, Python does not attach any particular meaning or
    significance to annotations. Left to its own, Python simply makes
    these expressions available as described in Accessing Function
    Annotations
    below.

    The only way that annotations take on meaning is when they are
    interpreted by third-party libraries. These annotation consumers
    can do anything they want with a function’s annotations. For
    example, one library might use string-based annotations to provide
    improved help messages, like so:

    def compile(source: "something compilable",
                filename: "where the compilable thing comes from",
                mode: "is this a single statement or a suite?"):
        ...
    

    Another library might be used to provide typechecking for Python
    functions and methods. This library could use annotations to
    indicate the function’s expected input and return types, possibly
    something like:

    def haul(item: Haulable, *vargs: PackAnimal) -> Distance:
        ...
    

    However, neither the strings in the first example nor the
    type information in the second example have any meaning on their
    own; meaning comes from third-party libraries alone.

  3. Following from point 2, this PEP makes no attempt to introduce
    any kind of standard semantics, even for the built-in types.
    This work will be left to third-party libraries.

    There is no worry that these libraries will assign semantics at
    random, or that a variety of libraries will appear, each with
    varying semantics and interpretations of what, say, a tuple of
    strings means. The difficulty inherent in writing annotation
    interpreting libraries will keep their number low and their
    authorship in the hands of people who, frankly, know what they’re
    doing.

Syntax

Parameters

Annotations for parameters take the form of optional expressions that
follow the parameter name. This example indicates that parameters
‘a’ and ‘c’ should both be an int, while parameter ‘b’ should
be a dict:

def foo(a: int, b: dict, c: int = 5):
    ...

In pseudo-grammar, parameters now look like identifier [:
expression] [= expression]
. That is, annotations always precede a
parameter’s default value and both annotations and default values are
optional. Just like how equal signs are used to indicate a default
value, colons are used to mark annotations. All annotation
expressions are evaluated when the function definition is executed.

Annotations for excess parameters (i.e., *args and **kwargs)
are indicated similarly. In the following function definition,
*args is flagged as a tuple of int, and **kwargs is
marked as a dict whose keys are strings and whose values are of type
str:

def foo(*args: int, **kwargs: str):
    ...

Note that, depending on what annotation-interpreting library you’re
using, the following might also be a valid spelling of the above:

def foo(*args: [int], **kwargs: {str: str}):
    ...

Only the first, however, has the BDFL’s blessing [3] as
the One Obvious Way.

Return Values

The examples thus far have omitted examples of how to annotate the
type of a function’s return value. This is done like so:

def sum(*args: int) -> int:
    ...

The parameter list can now be followed by a literal -> and a
Python expression. Like the annotations for parameters, this
expression will be evaluated when the function definition is executed.

The grammar for function definitions [11] is now:

decorator: '@' dotted_name [ '(' [arglist] ')' ] NEWLINE
decorators: decorator+
funcdef: [decorators] 'def' NAME parameters ['->' test] ':' suite
parameters: '(' [typedargslist] ')'
typedargslist: ((tfpdef ['=' test] ',')*
                ('*' [tname] (',' tname ['=' test])* [',' '**' tname]
                 | '**' tname)
                | tfpdef ['=' test] (',' tfpdef ['=' test])* [','])
tname: NAME [':' test]
tfpdef: tname | '(' tfplist ')'
tfplist: tfpdef (',' tfpdef)* [',']

Lambda

lambda’s syntax does not support annotations. The syntax of
lambda could be changed to support annotations, by requiring
parentheses around the parameter list. However it was decided
[12] not to make this change because:

  1. It would be an incompatible change.
  2. Lambda’s are neutered anyway.
  3. The lambda can always be changed to a function.

Accessing Function Annotations

Once compiled, a function’s annotations are available via the
function’s func_annotations attribute. This attribute is
a dictionary, mapping parameter names to an object representing
the evaluated annotation expression

There is a special key in the func_annotations mapping,
"return". This key is present only if an annotation was supplied
for the function’s return value.

For example, the following annotation:

def foo(a: 'x', b: 5 + 6, c: list) -> str:
    ...

would result in a func_annotation mapping of

{'a': 'x',
 'b': 11,
 'c': list,
 'return': str}

The return key was chosen because it cannot conflict with the name
of a parameter; any attempt to use return as a parameter name
would result in a SyntaxError.

func_annotations is an empty dictionary if no there are no
annotations on the function. func_annotations is always an empty
dictionary for functions created from lambda expressions.

Standard Library

pydoc and inspect

The pydoc module should display the function annotations when
displaying help for a function. The inspect module should change
to support annotations.

Relation to Other PEPs

Function Signature Objects [13]

Function Signature Objects should expose the function’s annotations.
The Parameter object may change or other changes may be warranted.

Implementation

A sample implementation for the syntax changes has been provided
[10] by Tony Lownds.

Rejected Proposals

  • The BDFL rejected the author’s idea for a special syntax for adding
    annotations to generators as being "too ugly" [4].
  • Though discussed early on ([5], [6]), including
    special objects in the stdlib for annotating generator functions and
    higher-order functions was ultimately rejected as being more
    appropriate for third-party libraries; including them in the
    standard library raised too many thorny issues.
  • Despite considerable discussion about a standard type
    parameterisation syntax, it was decided that this should also be
    left to third-party libraries. ([7],
    [8], [9])

Copyright

This document has been placed in the public domain.