derived metaclasses

2013.Apr.18

I was, again, planning to write about weakrefs and RAII, but that may have to wait until the weekend.

Daily blogging, by the way, is exhausting.

The NYC Python meetup had another fantastic meetup these evening at Peculier Pub. Those with stronger crowd estimation skills than I reckon that at least forty people came out! I can’t think of a better way to spend a Thursday evening than drinking beer with my fellow Python enthusiasts.


There is a principled ignorance we should embrace when employing inheritance in Python. That is, we should write the code for our derived entity in a way that it needs to know as little as possible about the base entity. In general, we should be able to write the derived entity without ever looking at the code of the base entity.

Python is fairly good in allowing us to do this (with a few strange edge-cases that I will blog about later.)

If I had the following base class:

1
2
class Base(object):
	pass # some code goes here

The it would be reasonable for me to write my derived object like this:

1
2
3
class Derived(Base):
	def __init__(self):
		Base.__init__(self)

(I’ve refrained from using super() in the above, because there are qualifications on its use, which I will blog about later. It’s probably a good idea to use super() in your code, so don’t let these examples distract you.)

In the above, we are following a very simple principle: * whenever we override some behaviour where we want to preserve (rather than replace) the intial behaviour, we call the corresponding function on our base case

In the above case, we don’t know if Base has a user-defined __init__ method, but we call it anyway, and the code just works.

So what if we had a metaclass?

As we know, there are two ways to write a metaclass: as a function (which, at the module-scope, can behave similarly to Python3’s __builtins__.__build_class__) or as a class derived from type. I’ll look at both formulations. I’ll look at the latter formulation first, since it’s a bit easier to understand.

If our metaclass is a class, we might override __new__ and delegate to type.__new__ for the actual construction of the type.

1
2
3
4
5
class Foo(object):
	class __metaclass__(type):
		def __new__(meta, name, bases, body):
			print 'Foo.__metaclass__({}, {}, {})'.format(name, bases, body)
			return type.__new__(meta, name, bases, body)

But what if Foo had a base class? We should follow the same principles as when we are working with methods that we may be overriding on the base class.

1
2
3
4
5
6
7
8
class Base(object):
	pass

class Derived(Base):
	class __metaclass__(type(Base)):
		def __new__(meta, name, bases, body):
			print 'Foo.__metaclass__({}, {}, {})'.format(name, bases, body)
			return type(Base).__new__(meta, name, bases, body)

Nicely, like __init__. this works as expected whether Base has a user-defined metaclass or not.

If there is a user-defined metaclass on Base, the code works as expected, calling the Derived metaclass and then the Base metaclass.

1
2
3
4
5
6
7
8
9
10
11
class Base(object):
	class __metaclass__(type(object)):
		def __new__(meta, name, bases, body):
			print 'Base.__metaclass__({}, {}, {})'.format(name, bases, body)
			return type(object).__new__(meta, name, bases, body)

class Derived(Base):
	class __metaclass__(type(Base)):
		def __new__(meta, name, bases, body):
			print 'Derived.__metaclass__({}, {}, {})'.format(name, bases, body)
			return type(Base).__new__(meta, name, bases, body)
# construction of type Base
Base.__metaclass__(Base, (<type 'object'>,), {'__module__': '__main__', '__metaclass__': <class '__main__.__metaclass__'>})

# construction of type Derived
Derived.__metaclass__(Derived, (<class '__main__.Base'>,), {'__module__': '__main__', '__metaclass__': <class '__main__.__metaclass__'>})

Now, what if our metaclass is a function?

If our metaclass is a function, we typically end it by calling type(name, bases, body) to actually construct the type.

1
2
3
4
class Foo(object):
	def __metaclass__(name, bases, body):
		print 'Foo.__metaclass__({}, {}, {})'.format(name, bases, body)
		return type(name, bases, body)

But what if Foo had a base class? We should follow the same principle as when we are working with methods that we may be overriding on the base class.

1
2
3
4
5
6
7
class Base(object):
	pass

class Derived(Base):
	def __metaclass__(name, bases, body):
		print 'Derived.__metaclass__({}, {}, {})'.format(name, bases, body)
		return type(Base)(name, bases, body)

The metaclass for any class is type(class), so the above lets us construct the Derived type object using the metaclass derived for it’s base class, which will defer to type(object) or just type.

Like in our __init__ example, the above code just works, even if Base does not actually have user defined metaclass.

If it does, the code behaves as we’d expect:

1
2
3
4
5
6
7
8
9
10
class Base(object):
	class __metaclass__(type(object)):
		def __new__(meta, name, bases, body):
			print 'Base.__metaclass__.__new__({}, {}, {})'.format(name, bases, body)
			return type(object).__new__(meta, name, bases, body)

class Derived(Base):
	def __metaclass__(name, bases, body):
		print 'Derived.__metaclass__({}, {}, {})'.format(name, bases, body)
		return type(Base)(name, bases, body)
# construction of type Base
Base.__metaclass__.__new__(Base, (<type 'object'>,), {'__module__': '__main__', '__metaclass__': <class '__main__.__metaclass__'>})

# construction of type Derived
Derived.__metaclass__(Derived, (<class '__main__.Base'>,), {'__module__': '__main__', '__metaclass__': <function __metaclass__ at 0x0>})
Base.__metaclass__.__new__(Derived, (<class '__main__.Base'>,), {'__module__': '__main__', '__metaclass__': <function __metaclass__ at 0x0>})

But wait a minute, what if Base’s metaclass is also a function?

1
2
3
4
5
6
7
8
9
class Base(object):
	def __metaclass__(name, bases, body):
		print 'Base.__metaclass__({}, {}, {})'.format(name, bases, body)
		return type(object)(name, bases, body)

class Derived(Base):
	def __metaclass__(name, bases, body):
		print 'Derived.__metaclass__({}, {}, {})'.format(name, bases, body)
		return type(Base)(name, bases, body)
# construction of type Base
Base.__metaclass__(Base, (<type 'object'>,), {'__module__': '__main__', '__metaclass__': <function __metaclass__ at 0x0>})

# construction of type Derived
Derived.__metaclass__(Derived, (<class '__main__.Base'>,), {'__module__': '__main__', '__metaclass__': <function __metaclass__ at 0x0>})

Shouldn’t we see Base.__metaclass__ called twice here?

No, we won’t, because when writing a metaclass function, you’re delegating the creation of the type to the base type object, so the type(Base) is type and not the metaclass.

This sounds awfully confusing, but it’s not, but in order to resolve this confusion, we’ll have to learn what a metaclass really is.

Next time: * what is a metaclass, really? * how do we properly delegate certain metaclass methods like __instancecheck__ and __subclasscheck__ in the absence of high-level access to the corresponding protocols (e.g., like getattr as a high-level access to a protocol that delegates to __getattr__)