apply and other strange decorators

2013.Apr.20

One of yesterday’s puzzles involved rebinding __builtins__.__build_class__ to something user-defined. This doesn’t happen often in real code.

(Prior to Python 2.3, one of the more common use-cases of rebinding names in __builtins__ was the overriding of __builtins__.__import__ to introduce custom behaviour when importing code. This use-case has gone away with the introduction of import and path hooks which I will blog about in depth later.)

The code looked something like this.

1
2
3
4
5
6
7
8
9
10
assert len([1,2,3,4]) == 4 # looks good

from functools import partial
@partial(lambda y,x:x(y), __builtins__.len)
def _(len):
	@partial(setattr, __builtins__, 'len')
	def _(x):
		return len(x)-1

assert len([1,2,3,4]) == 3 # what what what?

In this example, we’re overriding __builtins__.len with our own custom version. With this override, len(x) will give us one minus the actual length of x. Since we call the original len in our override, this code will work in every circumstance in which the original len works. Additionally, if we pass a bad argument to our custom len, we’ll get the correct Exception back.

Of course, any part of our system that relies on len working correctly will start to fail in strange ways. (Imagine what will happen to code that does if len(x): or code that blocks on while not len(x):!)

In general, there’s probably no reason we’d ever want to do this.

A less obfuscated way to write the above is:

1
2
3
4
5
6
7
# save the original len
__old_len__ = __builtins__.len

# override it with mylen
def mylen(x):
	return __old_len__(x)-1
__builtins__.len = mylen

Additional obfuscation in yesterday’s code comes from the use of a functools.partial to decorate the rebinding function.

A simpler example of the same concept (and an occassionally useful trick) is the use of __builtins__.apply as a decorator (available only in Python 2.)

As we know, a decorator is some callable that is immediately evaluated with the following class or function definition as its sole argument. The name of the class or function is rebound to the return value. (I’ll blog about what a decorator really is at some later date.)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# the decorating function
# in this case, doesn't do anything interesting
def decorate(f):
	return f

@decorate
def foo():
	return 'bar'

# desugars to (should be identical to):

def foo():
	return 'bar'
foo = decorate(foo)

apply is a very rarely used builtin. It's reminiscent of apply in Javascript: it's called with a function-value and some arguments, and it merely calls that function with those arguments. The arguments are supplied as a sequence of positionl arguments and a mapping of keyword arguments.

1
2
3
4
5
6
7
8
def f(x,y,z):
	print 'f({}, {}, {})'.format(x,y,z)
	return x+y+z

args = 1,2,3
assert f(*args) == apply(f, args) == apply(f, args, {})
kwargs = {'x': 1, 'y': 2, 'z': 3}
assert f(**kwargs) == apply(f, (), kwargs)

apply was introduced around Python 1.5. The documentation claims that it was introduced before we were able to sequence and keyword-unpack arguments using f(*args, **kwargs) syntax.

With the release of Python 2, we’ve been able to just use f(*args, **kwargs) syntax to call any function with an unpacked sequence of arguments and an unpacked mapping of keyword arguments.

A new syntax makes it more convenient to call a given function with a tuple of arguments and/or a dictionary of keyword arguments. In Python 1.5 and earlier, you’d use the apply() built-in function: apply(f, args, kw) calls the function f() with the argument tuple args and the keyword arguments in the dictionary kw. apply() is the same in 2.0, but thanks to a patch from Greg Ewing, f(*args, **kw) as a shorter and clearer way to achieve the same effect.

apply has been deprecated since Python 2.3 and is no longer available in Python 3.

Deprecated since version 2.3: Use function(*args, **keywords) instead of apply(function, args, keywords) (see Unpacking Argument Lists).

Using apply as a decorator has the effect of evaluating the following callable, then rebinding its name to the value produced.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# therefore
@apply
def foo():
	return 'bar'

# desugars to:
def foo():
	return 'bar'
foo = apply(foo)

# which is equivalent to:
def foo():
	return 'bar'
foo = foo()

Here’s an example of where you might use this trick in some almost useful fashion.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
from decimal import Decimal, localcontext

@apply
def pi(precision = 20, depth = 10):
	# ref: http://en.wikipedia.org/wiki/Pi#Continued_fractions  
	#      http://mathworld.wolfram.com/PiContinuedFraction.html
	with localcontext() as ctx:
		ctx.prec = precision
		denom = Decimal(0)
		for numer in xrange(2*depth-1, 0, -2):
			denom = Decimal(numer**2) / (6 + denom)
		return 3 + denom

print pi
assert round(pi,5) == 3.14159

In the above code, we define a function named pi. This function calculates the constant pi using continued fractions. When the interpreter evaluates this definition, it decorates this approximation function with the decorating function apply. The decorating function apply is immediately invoked with the approximating function as its argument:

1
2
pi = apply(pi)
pi = pi()

apply calls the approximating function. The result of this call is the value of the constant pi using a continued fraction of depth 10 (with division performed with exact precision up to 20 decimal digits.)

This result is bound to the name pi.

The original approximating function is removed from scope and garbage collected.

This formulation allows us to write out pi in the form of an approximating function that is evaluated exactly once (the moment the interpreter comes across its definition) and it henceforth bound to a (cached) value.

We accomplish this without cluttering our scope with the approximating function itself. To users of our code, it looks like we just statically defined pi to the resultant value.

Since there are no names bound to the approximating function, this function is also garbage collected, and so it is not just hidden from other code but actually totally unavailable.

pi is not a terribly useful example, but there may be other scientific or numeric constants that we want to capture in our programme as precomputed values that we want to compute just once on initialisation of our code and want to hold constant over the lifetime of our programme.

To continue this example…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
from itertools import count
from fractions import Fraction
from decimal import Decimal

# the constant `pi_fractions` is a mapping relating
#   {the numbers of digits of precision of pi:
#      first term in the sequence of Lange continued fractions that
#      has this many digits of precision}
# therefore, pi_fractions[n] should be the first of these continued fractions
#   that matches `pi` up to n digits
# e.g., pi_fractions[0] == 3/1
#       pi_fractions[1] == 22/7
#       pi_fractions[2] == 22/7
# ref: http://www.jstor.org/stable/2589152
@dict  # materialise into a mapping
@apply # initialise the generator
def pi_fractions():
	# ref: http://www.geom.uiuc.edu/~huberty/math5337/groupe/digits.html 
	pi_ref = '3.14159265' '3589793238' '4626433832' '7950288419' \
	         '7169399375' '1058209749' '4459230781' '6406286208' \
	         '9986280348' '2534211706' '7982148086' '5132823066' \
	         '4709384460' '9550582231' '7253594081' '2848111745' \
	         '0284102701' '9385211055' '5964462294' '8954930381' \
	         '9644288109' '7566593344' '6128475648' '2337867831'

	# approximation via continued fraction
	def pi(depth):
		if not depth:
			return Fraction(3, 1)
		frac = Fraction(1, 1)
		for numer in xrange(2*depth-1, 1, -2):
			frac = Fraction(numer**2, 6 + frac)
		return 3 + Fraction(1, 6 + frac)

	# successive approximations of pi
	#   using a textbook reference
	#   using a continued fraction calculation
	# pi_digits = (number of place, pi up to that place)
	# pi_calc = (depth of continued fraction, pi calculated to this depth)
	pi_digits = ((x,pi_ref[:x+2]) for x in xrange(1,len(pi_ref)))
	pi_calcs = ((x,pi(x)) for x in count())

	# first value
	depth, approx = next(pi_calcs)
	yield 0, approx

	for digits, ref in pi_digits:
		# TODO: find a faster converging continued fraction
		#         for pi, since this will start to take a LONG
		#         time to become accurate past a few digits 
		if digits > 8: break

		# calculate approximations of pi at increasing depths
		#   until we match the reference value 
		# may need to tweak precision to ensure we can tie out 
		while ref != '{:.{}f}'.format(Decimal(approx.numerator) /
		                              Decimal(approx.denominator), digits):
			depth, approx = next(pi_calcs)

		yield digits, approx

print pi_fractions
print pi_fractions[3] # the first continued fraction that gives us 3 digits
                      #   of precision (it's big!)

Using functools.partial works similarly to apply, but it allows us to provide arguments (and it’s also available in Python 3!)

1
2
3
4
5
6
7
8
9
from functools import partial

@partial(lambda f, x: f(x, y=2), x='why')
def foo(x, y):
	print 'foo({}, {})'.format(x, y)
	return x * y

assert foo == 'whywhy'
print foo

Next time: * why can’t we just use an anonymous function as a decorator? * how can we make Python allow us to use an anonymous function as a decorator?