December 12: Context Variables
As part of the Python Standard Library documentation traversal, we're following up on
yesterday's exhausting post about Operating System Services with
probably the shortest chapter yet: the
contextvars module. It's not cheating, it really is listed as its own
high-level documentation section!
So, alright, what are context variables? This is my probably flawed understanding: There is a general need for global
variables.¹ For example, flask (a Python web framework) has a "global variable"²
In regular old synchronous programming, you'd sometimes use
threading.local() to store global state. Variables put
into that state are available to everything in their thread, but anything accessing the "same" value in a different
thread would work on different data. That's all good when, for example, every incoming request is tied to a new thread
– but it fails if you want to recycle threads, or if you're using other sorts of concurrency, such as async³. Combine
same-thread concurrency with thread-locals and you get a horrible mess: if, for example, your context manager has a
state, and stores it in a thread-local variable, and then stores it there again from a different asynchronous call,
you will be very sad.
Enter stage left: Context variables. Context variables are values that live inside a specific context (basically a fancy
dictionary), and can be safely accessed and relied upon by everybody who is using this context. You instantiate them
var = ContextVar("var"). You access data with
var.get() and update it with
Asyncio has first-class support for context variables, and will copy and provide the correct context for you.
Everywhere else, you will need to handle your contexts with
copy_context(), which creates a new context from the
current one and returns a
Context object. You can then call methods specifically in that context by running
ctx.run(method). Any change the method makes to any context variable will be contained in the context obect. This
snippet helped me wrap my head around this:
>>> from contextvars import ContextVar, copy_context >>> animal = ContextVar("animal") >>> animal.set("cat") >>> def process_animal(): ... animal.set("dog") >>> ctx = copy_context() >>> animal.get() # Nothing has happened yet, 'cat' >>> ctx[animal] # so we have a lot of cats everywhere 'cat' >>> ctx.run(process_animal) # Calling the function in the context >>> animal.get() # We have not touched *this* variable 'cat' >>> ctx[animal] # … but we have changed it in this different context! 'dog'
Context variables are particularly encouraged for use in context managers whenever you need to store state in any way, to make sure no sneaky multiprocessing or async magic changes the values that you rely on.
¹ Arguably, and subject to opinionated debates: For example, flask is very into this pattern and Django is very much against it.
² Yes, yes.
³ Or any multi-threaded code, but I am not this tall.