sys.setrecursionlimit

2013.Apr.14

At office hours last week, sys.setrecursionlimit came up.

I mentioned that I had once seen code that looked like this:

1
2
3
4
5
6
7
8
9
10
11
from contextlib import contextmanager
from sys import getrecursionlimit, setrecursionlimit
@contextmanager
def recursionlimit(n=1000):
	rec_limit = getrecursionlimit()
	setrecursionlimit(n)
	yield
	setrecursionlimit(rec_limit)

with recursionlimit(1500):
	y = f(x)

It turns out that the author had written f in tail-recursive style. The data f operated on, denoted x, had grown just large enough to blow the stack and raise a RuntimeError. In lieu of refactoring f or convincing Guido to allow tail-call elimination, the author had decided to create a local context to bump up the stack limit.

I don’t know all the details behind the code or the data it operated on, so I will refrain from criticism of this approach. (I know the benefit of refactoring the code, but I don’t know the cost.)

This is not as unusual an approach as one might think. It is used in one place in the interpreter during exception matching:

1
2
3
4
5
6
7
8
9
        /* Temporarily bump the recursion limit, so that in the most
           common case PyObject_IsSubclass will not raise a recursion
           error we have to ignore anyway.  Don't do it when the limit
           is already insanely high, to avoid overflow */
        reclimit = Py_GetRecursionLimit();
        if (reclimit < (1 << 30))
            Py_SetRecursionLimit(reclimit + 5);
        res = PyObject_IsSubclass(err, exc);
        Py_SetRecursionLimit(reclimit);

For reference, the documentation tells us:

Set the maximum depth of the Python interpreter stack to limit. This limit prevents infinite recursion from causing an overflow of the C stack and crashing Python.

The highest possible limit is platform-dependent. A user may need to set the limit higher when she has a program that requires deep recursion and a platform that supports a higher limit. This should be done with care, because a too-high limit can lead to a crash.

It doesn’t tell us how low we can set the limit.

We are not allowed to set the recursion limit to 0.

1
2
3
4
5
>>> from sys import setrecursionlimit
>>> setrecursionlimit(0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: recursion limit must be positive
1
2
3
4
5
    if (new_limit <= 0) {
        PyErr_SetString(PyExc_ValueError,
                        "recursion limit must be positive");
        return NULL;
    }

However, we’re allowed to set it to something unusably low.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
>>> from sys import getrecursionlimit, setrecursionlimit
>>> rec_limit = getrecursionlimit()
>>> setrecursionlimit(1)
Error in sys.excepthook:
RuntimeError: maximum recursion depth exceeded

Original exception was:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: maximum recursion depth exceeded while calling a Python object
>>> # try to restore
>>> setrecursionlimit(rec_limit)

Error in sys.excepthook:
RuntimeError: maximum recursion depth exceeded

Original exception was:
RuntimeError: maximum recursion depth exceeded

Interestingly, since the REPL is itself written in Python and subject to this limit, there is no way to recover.

Next time: * why is this unrecoverable in the REPL? * what increments the stack depth counter, counting toward the limit? * what implications does this have on extension code (e.g., modules written in C)? * how do you recover from this in a script? * how would you have to write your code to be safe from this error? * what happens when you setrecursionlimit to something smaller than the current stack depth?