# apply and other strange decorators

## 2013.Apr.20

One of yesterday’s puzzles involved rebinding `__builtins__.__build_class__`

to something user-defined. This doesn’t happen often in real code.

(Prior to Python 2.3, one of the more common use-cases of rebinding names in `__builtins__`

was the overriding of `__builtins__.__import__`

to introduce custom behaviour when importing code. This use-case has gone away with the introduction of import and path hooks which I will blog about in depth later.)

The code looked something like this.

1
2
3
4
5
6
7
8
9
10

assert len([1,2,3,4]) == 4 # looks good
from functools import partial
@partial(lambda y,x:x(y), __builtins__.len)
def _(len):
@partial(setattr, __builtins__, 'len')
def _(x):
return len(x)-1
assert len([1,2,3,4]) == 3 # what what what?

In this example, we’re overriding `__builtins__.len`

with our own custom version. With this override, `len(x)`

will give us one minus the actual length of `x`

. Since we call the original `len`

in our override, this code will work in every circumstance in which the original `len`

works. Additionally, if we pass a bad argument to our custom `len`

, we’ll get the correct Exception back.

Of course, any part of our system that relies on `len`

working correctly will start to fail in strange ways. (Imagine what will happen to code that does `if len(x):`

or code that blocks on `while not len(x):`

!)

In general, there’s probably no reason we’d ever want to do this.

A less obfuscated way to write the above is:

1
2
3
4
5
6
7

# save the original len
__old_len__ = __builtins__.len
# override it with mylen
def mylen(x):
return __old_len__(x)-1
__builtins__.len = mylen

Additional obfuscation in yesterday’s code comes from the use of a `functools.partial`

to decorate the rebinding function.

A simpler example of the same concept (and an occassionally useful trick) is the use of `__builtins__.apply`

as a decorator (available only in Python 2.)

As we know, a decorator is some callable that is immediately evaluated with the following class or function definition as its sole argument. The name of the class or function is rebound to the return value. (I’ll blog about what a decorator *really* is at some later date.)

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# the decorating function
# in this case, doesn't do anything interesting
def decorate(f):
return f
@decorate
def foo():
return 'bar'
# desugars to (should be identical to):
def foo():
return 'bar'
foo = decorate(foo)

`apply`

is a very rarely used builtin. It's reminiscent of `apply`

in Javascript: it's called with a function-value and some arguments, and it merely calls that function with those arguments. The arguments are supplied as a sequence of positionl arguments and a mapping of keyword arguments.

1
2
3
4
5
6
7
8

def f(x,y,z):
print 'f({}, {}, {})'.format(x,y,z)
return x+y+z
args = 1,2,3
assert f(*args) == apply(f, args) == apply(f, args, {})
kwargs = {'x': 1, 'y': 2, 'z': 3}
assert f(**kwargs) == apply(f, (), kwargs)

`apply`

was introduced around Python 1.5. The documentation claims that it was introduced before we were able to sequence and keyword-unpack arguments using `f(*args, **kwargs)`

syntax.

With the release of Python 2, we’ve been able to just use `f(*args, **kwargs)`

syntax to call any function with an unpacked sequence of arguments and an unpacked mapping of keyword arguments.

A new syntax makes it more convenient to call a given function with a tuple of arguments and/or a dictionary of keyword arguments. In Python 1.5 and earlier, you’d use the apply() built-in function: apply(f, args, kw) calls the function f() with the argument tuple args and the keyword arguments in the dictionary kw. apply() is the same in 2.0, but thanks to a patch from Greg Ewing, f(*args, **kw) as a shorter and clearer way to achieve the same effect.

`apply`

has been deprecated since Python 2.3 and is no longer available in Python 3.

Deprecated since version 2.3: Use function(*args, **keywords) instead of apply(function, args, keywords) (see Unpacking Argument Lists).

Using `apply`

as a decorator has the effect of evaluating the following callable, then rebinding its name to the value produced.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# therefore
@apply
def foo():
return 'bar'
# desugars to:
def foo():
return 'bar'
foo = apply(foo)
# which is equivalent to:
def foo():
return 'bar'
foo = foo()

Here’s an example of where you might use this trick in some almost useful fashion.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

from decimal import Decimal, localcontext
@apply
def pi(precision = 20, depth = 10):
# ref: http://en.wikipedia.org/wiki/Pi#Continued_fractions
# http://mathworld.wolfram.com/PiContinuedFraction.html
with localcontext() as ctx:
ctx.prec = precision
denom = Decimal(0)
for numer in xrange(2*depth-1, 0, -2):
denom = Decimal(numer**2) / (6 + denom)
return 3 + denom
print pi
assert round(pi,5) == 3.14159

In the above code, we define a function named `pi`

. This function calculates the constant *pi* using continued fractions. When the interpreter evaluates this definition, it decorates this approximation function with the decorating function `apply`

. The decorating function `apply`

is immediately invoked with the approximating function as its argument:

1
2

pi = apply(pi)
pi = pi()

`apply`

calls the approximating function. The result of this call is the value of the constant *pi* using a continued fraction of depth 10 (with division performed with exact precision up to 20 decimal digits.)

This result is bound to the name `pi`

.

The original approximating function is removed from scope and garbage collected.

This formulation allows us to write out *pi* in the form of an approximating function that is evaluated exactly once (the moment the interpreter comes across its definition) and it henceforth bound to a (cached) value.

We accomplish this without cluttering our scope with the approximating function itself. To users of our code, it looks like we just statically defined `pi`

to the resultant value.

Since there are no names bound to the approximating function, this function is also garbage collected, and so it is not just hidden from other code but actually totally unavailable.

*pi* is not a terribly useful example, but there may be other scientific or numeric constants that we want to capture in our programme as precomputed values that we want to compute just once on initialisation of our code and want to hold constant over the lifetime of our programme.

To continue this example…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

from itertools import count
from fractions import Fraction
from decimal import Decimal
# the constant `pi_fractions` is a mapping relating
# {the numbers of digits of precision of pi:
# first term in the sequence of Lange continued fractions that
# has this many digits of precision}
# therefore, pi_fractions[n] should be the first of these continued fractions
# that matches `pi` up to n digits
# e.g., pi_fractions[0] == 3/1
# pi_fractions[1] == 22/7
# pi_fractions[2] == 22/7
# ref: http://www.jstor.org/stable/2589152
@dict # materialise into a mapping
@apply # initialise the generator
def pi_fractions():
# ref: http://www.geom.uiuc.edu/~huberty/math5337/groupe/digits.html
pi_ref = '3.14159265' '3589793238' '4626433832' '7950288419' \
'7169399375' '1058209749' '4459230781' '6406286208' \
'9986280348' '2534211706' '7982148086' '5132823066' \
'4709384460' '9550582231' '7253594081' '2848111745' \
'0284102701' '9385211055' '5964462294' '8954930381' \
'9644288109' '7566593344' '6128475648' '2337867831'
# approximation via continued fraction
def pi(depth):
if not depth:
return Fraction(3, 1)
frac = Fraction(1, 1)
for numer in xrange(2*depth-1, 1, -2):
frac = Fraction(numer**2, 6 + frac)
return 3 + Fraction(1, 6 + frac)
# successive approximations of pi
# using a textbook reference
# using a continued fraction calculation
# pi_digits = (number of place, pi up to that place)
# pi_calc = (depth of continued fraction, pi calculated to this depth)
pi_digits = ((x,pi_ref[:x+2]) for x in xrange(1,len(pi_ref)))
pi_calcs = ((x,pi(x)) for x in count())
# first value
depth, approx = next(pi_calcs)
yield 0, approx
for digits, ref in pi_digits:
# TODO: find a faster converging continued fraction
# for pi, since this will start to take a LONG
# time to become accurate past a few digits
if digits > 8: break
# calculate approximations of pi at increasing depths
# until we match the reference value
# may need to tweak precision to ensure we can tie out
while ref != '{:.{}f}'.format(Decimal(approx.numerator) /
Decimal(approx.denominator), digits):
depth, approx = next(pi_calcs)
yield digits, approx
print pi_fractions
print pi_fractions[3] # the first continued fraction that gives us 3 digits
# of precision (it's big!)

Using `functools.partial`

works similarly to `apply`

, but it allows us to provide arguments (and it’s also available in Python 3!)

1
2
3
4
5
6
7
8
9

from functools import partial
@partial(lambda f, x: f(x, y=2), x='why')
def foo(x, y):
print 'foo({}, {})'.format(x, y)
return x * y
assert foo == 'whywhy'
print foo

Next time: * why can’t we just use an anonymous function as a decorator? * how can we make Python allow us to use an anonymous function as a decorator?