lambda decorators

2013.Apr.21

In yesterday’s post, we looked at using strange functions as decorators.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
from datetime import date

# a mapping of the locale-specific month short names
#   to the month number [1,12]
# {month number: month short name}
@apply
def months():
	return {x: date(2000,x,1).strftime('%b') for x in xrange(1,12+1)}

# set of the locale-specific weekday short names
@set
@apply
def weekdays():
	for x in xrange(1,7+1):
		yield date(2000,1,x).strftime('%a')

print months, weekdays

We might ask: what can’t we use as a decorator?

It turns out that the allowed syntax for decorators is fairly limited, and these limits are intentional.

Around the time that decorators were introduced into Python, Guido made an executive decision to disallow certain constructions following the @.

The @ in decorator syntax can be followed only by a dotted name which can (optionally) itself be invoked as a callable with arguments.

No other expression can follow the @.

As a result, only the following are legal:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# legal
@decorator # some callable
def foo():
	pass

# legal
@bar.decorator # some dotted name/attribute lookup that gives a callable
def foo():
	pass

# legal
@decorator() # some callable (with args) that returns a callable
def foo():
	pass

# legal
@bar.decorator() # combination of the above 
def foo():
	pass

The following are among the disallowed:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# illegal
@bar().decorator # can't have a function call beofre the end
def foo():
	pass

# illegal
@lambda x: x # no lambdas!
def foo():
	pass

# illegal
@decorators[0] # no other expression syntax, like item-getting
def foo():
	pass

# illegal
@decorator()() # only a single evaluation is allowed
def foo():
	pass

# illegal
@(decorator1 if True else decorator2) # no expression syntax, like if/else
def foo():
	pass

These are probably reasonable restrictions. But how can we lift them? How do we get the interpreter to accept these constructions as legal?

Well, it turns out to be surprisingly easy.

We need to make only three, small modifications.

To do this, let’s start with the latest Python 2 release. (Patching Python 3 isn’t much different.)

First, let’s look at the file Grammar/Grammar. Grammar/Grammar is an EBNF description of the Python language grammar.

Grammar/Grammar explains its contents:

Input is a grammar in extended BNF (using * for repetition, + for
at-least-once repetition, [] for optional parts, | for alternatives and
() for grouping).  This has already been parsed and turned into a parse
tree.

Each rule is considered as a regular expression in its own right.
It is turned into a Non-deterministic Finite Automaton (NFA), which
is then turned into a Deterministic Finite Automaton (DFA), which is then
optimized to reduce the number of states.  See [Aho&Ullman 77] chapter 3,
or similar compiler books (this technique is more often used for lexical
analyzers).

The DFA's are used by the parser as parsing tables in a special way
that's probably unique.  Before they are usable, the FIRST sets of all
non-terminals are computed.

This file is used as input to the parser generator to actually create the parser that is used to parse code in the interpreter.

From the Makefile:

$(GRAMMAR_H): $(GRAMMAR_INPUT) $(PGENSRCS)
		@$(MKDIR_P) Include
		$(MAKE) $(PGEN)
		$(PGEN) $(GRAMMAR_INPUT) $(GRAMMAR_H) $(GRAMMAR_C)

If we search though Grammar/Grammar for the decorator syntax, we’ll find the following two lines:

decorator: '@' dotted_name [ '(' [arglist] ')' ] NEWLINE

dotted_name: NAME ('.' NAME)*

These say that we’re allowed to follow the @ symbol with a list of any length of dotted names (i.e., attribute name lookups, valid identifiers separated by dots.) At the very end of this qualifier, we’re allowed a single function invocation, optionally providing arguments.

To free up the Grammar, let’s change this from dotted_name to testlist. testlist is the syntax node type for expressions. With this change, we can support any arbitrary, valid expression after the @.

This should be valid, as long as that expression returns a callable which can decorate the following, defined function.

Our change looks something like (in patch format):

--- Grammar/Grammar 2013-04-06 10:02:25.000000000 -0400
+++ Grammar/Grammar 2013-04-23 00:32:46.087990352 -0400
@@ -19,7 +19,7 @@ single_input: NEWLINE | simple_stmt | co
 file_input: (NEWLINE | stmt)* ENDMARKER
 eval_input: testlist NEWLINE* ENDMARKER
 
-decorator: '@' dotted_name [ '(' [arglist] ')' ] NEWLINE
+decorator: '@' testlist NEWLINE
 decorators: decorator+
 decorated: decorators (classdef | funcdef)
 funcdef: 'def' NAME parameters ':' suite

Now, if we make && make test, we’ll see the parser automatically regengerated as a result of our change to the grammar. Fantastic!

However, we should also two test failures: 1. Lib/test/test_decorators.py 2. Lib/test/test_parser.py

The first test is easy to fix. The code looks like:

# Test syntax restrictions - these are all compile-time errors:
#
for expr in [ "1+2", "x[3]", "(1, 2)" ]:
    # Sanity check: is expr is a valid expression by itself?
    compile(expr, "testexpr", "exec")

    codestr = "@%s\ndef f(): pass" % expr
    self.assertRaises(SyntaxError, compile, codestr, "test", "exec")

This just checks that certain constructions are flagged as illegal when parsing. Since we want to consider these as valid constructions, we don’t want to flag these errors. We can remove these tests entirely.

Our change looks something like (in patch format):

--- Lib/test/test_decorators.py 2013-04-06 10:02:30.000000000 -0400
+++ Lib/test/test_decorators.py 2013-04-20 11:48:53.394309627 -0400
@@ -152,15 +152,6 @@ class TestDecorators(unittest.TestCase):
         self.assertEqual(counts['double'], 4)
 
     def test_errors(self):
-        # Test syntax restrictions - these are all compile-time errors:
-        #
-        for expr in [ "1+2", "x[3]", "(1, 2)" ]:
-            # Sanity check: is expr is a valid expression by itself?
-            compile(expr, "testexpr", "exec")
-
-            codestr = "@%s\ndef f(): pass" % expr
-            self.assertRaises(SyntaxError, compile, codestr, "test", "exec")
-
         # You can't put multiple decorators on a single line:
         #
         self.assertRaises(SyntaxError, compile,

The second test failure (test_parser.py) occurs when validating a roundtrip generation of the syntax tree. The code for this test is a bit spread out, so I won’t excerpt it here.

When validating, the parser expects a dotted_name as a child of the syntax tree, but we’re now providing a testlist.

This can be fixed with the following simple patch to the parser to tell it to expect a testlist instead of a dotted_name.

Our change looks something like (in patch format):

--- Modules/parsermodule.c  2013-04-06 10:02:37.000000000 -0400
+++ Modules/parsermodule.c  2013-04-20 17:35:50.178307543 -0400
@@ -2621,7 +2621,7 @@ validate_decorator(node *tree)
     ok = (validate_ntype(tree, decorator) &&
           (nch == 3 || nch == 5 || nch == 6) &&
           validate_at(CHILD(tree, 0)) &&
-          validate_dotted_name(CHILD(tree, 1)) &&
+          validate_testlist(CHILD(tree, 1)) &&
           validate_newline(RCHILD(tree, -1)));
 
     if (ok && nch != 3) {

That’s it!

If we make && make test, we’ll see no more errors!

Not bad: only three lines of changes across only three files to change a fairly substantial part of the language! (I’ll blog more about the kinds of exploratory language modifications we can make in Python.)

So, let’s try it out.

>>> # these decorators don't do anything useful
... #   but it shows that we can now use lambdas or any
... #   other arbitrary, valid expression
...

>>> @lambda f: lambda *args, **kwargs: (f(*args, **kwargs), f(*args,**kwargs))
... def foo():
...   return 'foo'
... 
>>> foo()
('foo', 'foo')

>>> from functools import wraps
>>> def double(f):
...   @wraps(f)
...   def dec(*args, **kwargs):
...       return f(*args, **kwargs), f(*args, **kwargs)
...   return dec
... 
>>> def triple(f):
...   @wraps(f)
...   def dec(*args, **kwargs):
...       return f(*args, **kwargs), f(*args, **kwargs), f(*args, **kwargs)
...   return dec
... 

>>> @[double, triple][0]
... def foo():
...   return 'foo'
... 
>>> foo()
('foo', 'foo')

>>> @double if True else triple
... def foo():
...    return 'foo'
... 
>>> foo()
('foo', 'foo')

Of course, we’re still restricted from using statements (e.g., class definitions, function definitions, print in Python 2, assignments) and the expression that we use must return a callable.

Other than that, however, we can do anything!

Bear in mind that we could have accomplished the same thing without any modifications to the interpreter with a simple helper below—but where’s the fun in that?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
_ = lambda expr: expr

@_(lambda f: lambda *args, **kwargs: (f(*args, **kwargs), f(*args, **kwargs)))
def foo():
	return 'foo'

assert foo() == ('foo', 'foo')

from functools import wraps
def double(f):
	@wraps(f)
	def dec(*args, **kwargs):
		return f(*args, **kwargs), f(*args, **kwargs)
	return dec

def triple(f):
	@wraps(f)
	def dec(*args, **kwargs):
		return f(*args, **kwargs), f(*args, **kwargs), f(*args, **kwargs)
	return dec

@_([double, triple][0])
def foo():
	return 'foo'

assert foo() == ('foo', 'foo')

@_(double if True else triple)
def foo():
	return 'foo'

assert foo() == ('foo', 'foo')

Next time: * what kind of strange things can we do with this? * what are decorators, really? (and why was this change so easy?)