I was surprised with the with open("somefile"): syntax in Python. This is useful for automatically freeing resources, such as opening files:
with open("some_file") as f:
print(f.read())
# f is automatically closed at the end of the with statement and cannot be used:
print(f.tell())
If we run this code, you’ll have an error because the file is closed and that’s what is expected:
Traceback (most recent call last):
File "<stdin>", line xy, in <module>
ValueError: I/O operation on closed file.
There are some more examples in the contextlib package, for instance in order to acquire a database connection for a query.
It turns out this pattern is called a context manager and you can write your own. All you have to do is implement __enter__ and __exit__.
Here is an example where we temporarilly increase the allowed memory limit and recursion depth, then we restore it to its initial value when it’s over.
import resource, sys
# Custom context manager for max recursion depth and memory usage
# https://docs.python.org/3/reference/datamodel.html#context-managers
class ResourceLimit:
def __init__(self, recursion_limit, stack_limit):
self.recursion_limit = recursion_limit
self.stack_limit = stack_limit
def __enter__(self):
self.old_recursion_limit = sys.getrecursionlimit()
self.old_memory_limit = resource.getrlimit(resource.RLIMIT_STACK)
# https://docs.python.org/3/library/sys.html#sys.setrecursionlimit
# https://docs.python.org/3/library/resource.html#resource.setrlimit
sys.setrecursionlimit(self.recursion_limit)
resource.setrlimit(resource.RLIMIT_STACK, (self.stack_limit, -1))
def __exit__(self, type, value, tb):
sys.setrecursionlimit(self.old_recursion_limit)
resource.setrlimit(resource.RLIMIT_STACK, self.old_memory_limit)
print("before")
print(sys.getrecursionlimit())
print(resource.getrlimit(resource.RLIMIT_STACK))
# We may reach 1_000_000 levels of recursion
# and we have up to 512 MB of memory
with ResourceLimit(1_000_000, 512 * 1024 * 1024):
print("during")
print(sys.getrecursionlimit())
print(resource.getrlimit(resource.RLIMIT_STACK))
complex_function_that_needs_a_lot_of_ram_and_uses_recursion_a_lot()
print("after")
print(sys.getrecursionlimit())
print(resource.getrlimit(resource.RLIMIT_STACK))
The resources are increased then restored, as you can see in the output:
before
1000
(8388608, -1)
during
1000000
(536870912, -1)
after
1000
(8388608, -1)
This is a contrieved example (everything goes back to normal at the end of the program anyway), but this can be useful for chaining some problems that may require extra resources.