December 12: Context Variables

As part of the Python Standard Library documentation traversal, we're following up on yesterday's exhausting post about Operating System Services with probably the shortest chapter yet: the contextvars module. It's not cheating, it really is listed as its own high-level documentation section!

So, alright, what are context variables? This is my probably flawed understanding: There is a general need for global variables.¹ For example, flask (a Python web framework) has a "global variable"² request.

In regular old synchronous programming, you'd sometimes use threading.local() to store global state. Variables put into that state are available to everything in their thread, but anything accessing the "same" value in a different thread would work on different data. That's all good when, for example, every incoming request is tied to a new thread – but it fails if you want to recycle threads, or if you're using other sorts of concurrency, such as async³. Combine same-thread concurrency with thread-locals and you get a horrible mess: if, for example, your context manager has a state, and stores it in a thread-local variable, and then stores it there again from a different asynchronous call, you will be very sad.

Enter stage left: Context variables. Context variables are values that live inside a specific context (basically a fancy dictionary), and can be safely accessed and relied upon by everybody who is using this context. You instantiate them with var = ContextVar("var"). You access data with var.get() and update it with var.set()

usage

Asyncio has first-class support for context variables, and will copy and provide the correct context for you. Everywhere else, you will need to handle your contexts with copy_context(), which creates a new context from the current one and returns a Context object. You can then call methods specifically in that context by running ctx.run(method). Any change the method makes to any context variable will be contained in the context obect. This snippet helped me wrap my head around this:

>>> from contextvars import ContextVar, copy_context

>>> animal = ContextVar("animal")
>>> animal.set("cat")

>>> def process_animal():
...    animal.set("dog")

>>> ctx = copy_context()
>>> animal.get()  # Nothing has happened yet, 
'cat'
>>> ctx[animal]  # so we have a lot of cats everywhere
'cat'

>>> ctx.run(process_animal)  # Calling the function in the context

>>> animal.get()  # We have not touched *this* variable
'cat'
>>> ctx[animal]  # … but we have changed it in this different context!
'dog'

Context variables are particularly encouraged for use in context managers whenever you need to store state in any way, to make sure no sneaky multiprocessing or async magic changes the values that you rely on.

¹ Arguably, and subject to opinionated debates: For example, flask is very into this pattern and Django is very much against it.

² Yes, yes.

³ Or any multi-threaded code, but I am not this tall.