Hi, Future Me! It seems that you've forgotten once again how Django's @page_cache decorator function works. But this
time, I'm here to tell you what you (or rather, I) worked out in the past. So here it is: @page_cache from the ground
up.
Together with a friend, I’m running the Mastodon/Fediverse instance chaos.social. We’ve been at it for nearly five years now, and I feel like it’s time to reflect on the good, the bad, the ugly. Please note that you'll be reading an edited version of this post with most of the cursing removed, so please be understanding of the bits that I kept – they are load-bearing.
This is an edited transcript of the keynote I delivered at PyCon UK in 2019. I'm still grateful to the organisers, and I still treasure my memories of visiting Cardiff for the first time that year. I've edited the transcript only to make it readable, fixing sentences and removing filler words – but the content and structure remains unchanged. I also left in audience reactions, for flair (and because I'm a bit proud to have made people laugh). See the end notes for corrections and other remarks – you can also see the full slides including speaker notes here.
Now we get to talk about a completely trivial topic, barely worth mentioning: Concurrent execution, including
threading, multiprocessing, concurrent, subprocess, sched and queue.
Now, you might say, "wait a second, it is not December 13th!", and … you would be right. Life got in the way and I had to spend a week figuring some things out, and pushing out a blog post about concurrency in Python did not quite fit in with that. Now all the things are figured out, and I'm continuing the Traversal of the Python Standard Library Documentation.
As part of the Python Standard Library documentation traversal, we're following up on
yesterday's exhausting post about Operating System Services with
probably the shortest chapter yet: the contextvars module. It's not cheating, it really is listed as its own
high-level documentation section!
After some restful days, today's Python Standard Library Traversal follows up yesterday's
Cryptographic Service Modules with all the generic OS
interaction modules. That is os (but not os.path, we did that a week
ago), io, time, argparse, getopt, logging and its
submodules, getpass, curses and submodules, platform, errno and ctypes. Strap in!
os.startfile() acts like double-clicking the file.argparse is honestly not as bad as I remembered it. Stockholm syndrome?logger.getChild() is like calling getLogger() on the full target name, very useful when you use stand-in values
like __name__.logger.handlers.TimedRotatingFileHandler and logger.handlers.RotatingFileHandler handle log file rotation, nice!logger.handlers.HTTPHandler defaults to secure=False.%-style string formatting in logging formatters to str.format() or string.Template.platform module.Shhh! 🤫 As part of the Python Standard Library traversal, after yesterday's
look at file format modules, today is going to be all about secrets.
Well, all about secrets and hmac and hashlib, to be precise.
hashlib.secrets module exists and is suitable for (reasonably) strong randomness, eg for generating long secret URLs.Combining the commonplace and the weirdly specific: As part of the Python Standard Library
traversal, after yesterday's short post about data
compression and archiving modules, today we're going to
look at Python file format modules. Those include csv, configparser, netrc, xdrlib and the forgettable
plistlib.
csv dialect is extremely easy. I wonder how cursed you can make it.configparser has an extended interpolation mode that allows you to refer to values from other sections like
%{OtherSection:other_value}
configparser, like the delimiters, comment prefix, strict mode, enable
multiline strings with empty lines, permit empty/flag values, …csv in a different documentation block from json and xml?netrc FTP configuration format exists, and requires its own stdlib support module.xdr file format exists, and at least the Python interface sounds pretty annoying.plist Apple format exists, and it sounds not that bad.As part of the Python Standard Library
traversal, after yesterday's short post about data
persistence, today we're going to look at Python data compression
modules. There are not too many of them, but definitely more than I knew: zlib, gzip, bz2, lzma, zipfile,
tarfile.
zipfile module has a CLI interface for compression, extraction and listing of archives.tarfile.w:bz2.Strap in, this is a long one: As part of the Python Standard Library
traversal, after yesterday's short post about binary
data services, today we're going to look at Python data type modules.
There's a lot of them: datetime, calendar, collections, heapq, bisect, array, weakref, types, copy,
pprint, reprlib and enum.
shelve to store pickled objects in a persistent key-value store.dbm library is a thing that exists and has Python standard library support.There's no place where "one obvious way to do things" fails as much as it does with file and OS interaction. As part of
the Python Standard Library traversal, after
yesterday's post about functional programming modules, today
we're going to look at the file and directory access modules. Yes, all of them: pathlib, os.path, fileinput,
stat, filecmp, tempfile, glob, fnmatch, linecache and shutil.
pathlib and os.path is bordering on hilarious. One obvious way indeed.fileinput is really weird.stat allows you to query and extract results of os.stat()
shutil.diskusage() returns total, used and free bytes for a given directory.shutil.which() looks up executables (platform independent which)shutil.make_archive supports zip, tar, gztar, bztar and xtar out of the box.A short entry for a change: As part of the Python Standard Library
traversal, after yesterday's short post about
Python's numeric and mathematical modules, I'm taking a look
at itertools, functools and operator.
itertools functions.functools contains a cached_property decorator.lru_cache does.singledispatch.operator functions for performing in-place operations like +=.operator.attrgetter can retrieve multiple and nested attributes.operator.itemgetter can retrieve multiple items.Today is a bit less bad than yesterday: As part of the Python Standard Library traversal, after yesterday's extensive post about data types, today we're going to look at Python modules that take care of numbers and maths and make us very happy because we don't need to implement them ourselves. Or wait for NumPy to install.
numbers module contains abstract base classes so you can check for features of given numbers using isinstance.tau and against pi. Good documentation.cmath provides mathematical operations and functions for complex numbers – it exists separately from math so that
people aren't surprised when they run math.sqrt(-1). 🤣fractions.limit_denominator() allows you to take a float and turn it into the number it was meant to be. Go team
fractions.decimal is singular and fractions is plural will never cease to confuse me.random.gauss() and random.normalvariate() ???statistics.NormalDist creates a normal distribution either from a set of samples or the mean and a standard
deviation.Strap in, this is a long one: As part of the Python Standard Library
traversal, after yesterday's short post about binary
data services, today we're going to look at Python data type modules.
There's a lot of them: datetime, calendar, collections, heapq, bisect, array, weakref, types, copy,
pprint, reprlib and enum.
datetime.datetime.time(), use timetz().calendar in a few days.collections is good and underused.maxlen attribute and will discard items past that length.weakref.finalize(obj, callback)
list and dict subclasses, not the original types themselves.types.coroutine() can turn a generator function into an awaitable coroutine.As part of the Python Standard Library traversal,
today we're following up yesterday's text processing services with
the binary data services: struct and codecs.
codecs.encode("foo", "rot13") 😍struct.As part of the Python Standard Library traversal, today, we're going through the Python text processing services: string, re, difflib, textwrap, unicodedata, stringprep, readline, rlcompleter.
string.Template is kind of nice for user-facing string substitution.string.capwords() runs str.capitalize() on every word in a string.difflib.get_close_matches can help you catch user typos.re.split is like "".split, but for regular expressions.re.sub can take a function instead of a replacement text.textwrap: textwrap is good, use it! Particularly text shortening and indent/dedent
functionality is a really nice add-on.After yesterday's built-in functions, today's entry in the traversal of the Python Standard Library docs takes us to the built-in constants, types and exceptions.
The things I learned or found most interesting in this post:
divmod exists, oops.string.casefold() for case-insensitive comparison, and expandtab() to replace tabs with spaces, without
reverse operation, mwahaha.[2:10:3]) (waat)memoryview, which lets you access and manipulate bytes and bytearrays directly by exposing
the raw integers.my_set.union() you can actually use any iterable as argument (as opposed to my_set | other_set
where other_set really needs to be a set)This post is part of my series on Traversing the Python Standard Library.
Built-in functions are included in the standard library and are always available with no need for imports. There are 69 (nice) of them. I've rearranged them from the Python documentation order (alphabetical) into groups – this was short enough to do a detailed breakdown, the following days will be more of a summary.
These are the things I did not know or had forgotten:
open() takes an errors argument that you can set to surrogateescape, which allows you to process and even
roundtrip encoding errors in files.iter() can take two arguments and then calls the first one repeatedly until its result equals the second argument,
good for chunked reading.round(), when it could round either way, chooses the even number, so both round(1.5) and round(2.5) is 2.I like Python.
I've been a professional programmer for at least five years now, depending on how you count, and I've spent a majority of that time with Python. I like it. It's comfortable, it's fun, it doesn't get in my way too much most of the time, it has a rich ecosystem filled with good people who make work pleasant, and many of who have turned into friends over the years.
And yet … I have also never gotten over a slightly imposter-syndrome tinged sense of unease around the language. Sure, I've been using it for close to a decade, and sure, I've been putting in time and effort to learn new things and avoid bad patterns. When called upon to relay my experience level, I call myself fluent or experienced or proficient. And yet …
If your Django project runs into exceptions in production, Django can email you a complete stacktrace, and all the context information you'd usually see in development mode. As of Django 3.0, you can tweak the content of these emails (and Django 3.1 made it a bit easier for us, so this blog post assumes Django 3.1 or later releases).