How Django's @page_cache works

Hi, Future Me! It seems that you've forgotten once again how Django's @page_cache decorator function works. But this time, I'm here to tell you what you (or rather, I) worked out in the past. So here it is: @page_cache from the ground up.

Read more…

while history: continue

This is an edited transcript of the keynote I delivered at PyCon UK in 2019. I'm still grateful to the organisers, and I still treasure my memories of visiting Cardiff for the first time that year. I've edited the transcript only to make it readable, fixing sentences and removing filler words – but the content and structure remains unchanged. I also left in audience reactions, for flair (and because I'm a bit proud to have made people laugh). See the end notes for corrections and other remarks – you can also see the full slides including speaker notes here.

Read more…

December 13: Python Concurrent Execution Modules

Now we get to talk about a completely trivial topic, barely worth mentioning: Concurrent execution, including threading, multiprocessing, concurrent, subprocess, sched and queue.

Now, you might say, "wait a second, it is not December 13th!", and … you would be right. Life got in the way and I had to spend a week figuring some things out, and pushing out a blog post about concurrency in Python did not quite fit in with that. Now all the things are figured out, and I'm continuing the Traversal of the Python Standard Library Documentation.

Highlights

  • Oh god Popen has so many arguments.
  • Python ships a "general purpose event scheduler"

Read more…

December 11: Python Operating System Services

After some restful days, today's Python Standard Library Traversal follows up yesterday's Cryptographic Service Modules with all the generic OS interaction modules. That is os (but not os.path, we did that a week ago), io, time, argparse, getopt, logging and its submodules, getpass, curses and submodules, platform, errno and ctypes. Strap in!

Highlights

  • On Windows, os.startfile() acts like double-clicking the file.
  • There are way too many ways to start new processes in Python.
  • "seconds since epoch" usually excludes leap seconds.
  • argparse is honestly not as bad as I remembered it. Stockholm syndrome?
  • logger.getChild() is like calling getLogger() on the full target name, very useful when you use stand-in values like __name__.
  • logger.handlers.TimedRotatingFileHandler and logger.handlers.RotatingFileHandler handle log file rotation, nice!
  • logger.handlers.HTTPHandler defaults to secure=False.
  • You can change the default %-style string formatting in logging formatters to str.format() or string.Template.
  • You can query the platform you are running on with the platform module.

Read more…

December 9: Python File Format Modules

Combining the commonplace and the weirdly specific: As part of the Python Standard Library traversal, after yesterday's short post about data compression and archiving modules, today we're going to look at Python file format modules. Those include csv, configparser, netrc, xdrlib and the forgettable plistlib.

Highlights

  • Writing your own csv dialect is extremely easy. I wonder how cursed you can make it.
  • configparser has an extended interpolation mode that allows you to refer to values from other sections like %{OtherSection:other_value}
  • You can configure lots of things about configparser, like the delimiters, comment prefix, strict mode, enable multiline strings with empty lines, permit empty/flag values, …
  • Why on earth is csv in a different documentation block from json and xml?
  • I learnt that the netrc FTP configuration format exists, and requires its own stdlib support module.
  • I learnt that xdr file format exists, and at least the Python interface sounds pretty annoying.
  • I learnt that plist Apple format exists, and it sounds not that bad.

Read more…

December 8: Pyton Data Compression and Archiving Modules

As part of the Python Standard Library traversal, after yesterday's short post about data persistence, today we're going to look at Python data compression modules. There are not too many of them, but definitely more than I knew: zlib, gzip, bz2, lzma, zipfile, tarfile.

Highlights

  • Now I know that Python supports six different main compression protocols out of the box.
  • The zipfile module has a CLI interface for compression, extraction and listing of archives.
  • Same for tarfile.
  • When opening a tarfile, you can pass the intended compression algorithm with the file mode, like w:bz2.

Read more…

December 7: Python Data Persistence Modules

Strap in, this is a long one: As part of the Python Standard Library traversal, after yesterday's short post about binary data services, today we're going to look at Python data type modules. There's a lot of them: datetime, calendar, collections, heapq, bisect, array, weakref, types, copy, pprint, reprlib and enum.

Highlights

  • You can use shelve to store pickled objects in a persistent key-value store.
  • The Unix dbm library is a thing that exists and has Python standard library support.

Read more…

December 6: Python File and Directory Access

There's no place where "one obvious way to do things" fails as much as it does with file and OS interaction. As part of the Python Standard Library traversal, after yesterday's post about functional programming modules, today we're going to look at the file and directory access modules. Yes, all of them: pathlib, os.path, fileinput, stat, filecmp, tempfile, glob, fnmatch, linecache and shutil.

Highlights

  • The 80% overlap between pathlib and os.path is bordering on hilarious. One obvious way indeed.
  • fileinput is really weird.
  • stat allows you to query and extract results of os.stat()
  • shutil.diskusage() returns total, used and free bytes for a given directory.
  • shutil.which() looks up executables (platform independent which)
  • shutil.make_archive supports zip, tar, gztar, bztar and xtar out of the box.

Read more…

December 5: Python Functional Programming Modules

A short entry for a change: As part of the Python Standard Library traversal, after yesterday's short post about Python's numeric and mathematical modules, I'm taking a look at itertools, functools and operator.

Highlights

  • There really are a lot of itertools functions.
  • functools contains a cached_property decorator.
  • I finally know what lru_cache does.
  • Apparently Python has a type of function overloading, by type of first argument, with singledispatch.
  • There are operator functions for performing in-place operations like +=.
  • operator.attrgetter can retrieve multiple and nested attributes.
  • operator.itemgetter can retrieve multiple items.

Read more…

December 4: Python Numeric and Mathematical Modules

Today is a bit less bad than yesterday: As part of the Python Standard Library traversal, after yesterday's extensive post about data types, today we're going to look at Python modules that take care of numbers and maths and make us very happy because we don't need to implement them ourselves. Or wait for NumPy to install.

Highlights

  • The numbers module contains abstract base classes so you can check for features of given numbers using isinstance.
  • The Python documentation takes a stand for tau and against pi. Good documentation.
  • cmath provides mathematical operations and functions for complex numbers – it exists separately from math so that people aren't surprised when they run math.sqrt(-1). 🤣
  • fractions.limit_denominator() allows you to take a float and turn it into the number it was meant to be. Go team fractions.
  • The fact that decimal is singular and fractions is plural will never cease to confuse me.
  • Apparently there's both random.gauss() and random.normalvariate() ???
  • statistics.NormalDist creates a normal distribution either from a set of samples or the mean and a standard deviation.

Read more…

December 3: Python Data Types

Strap in, this is a long one: As part of the Python Standard Library traversal, after yesterday's short post about binary data services, today we're going to look at Python data type modules. There's a lot of them: datetime, calendar, collections, heapq, bisect, array, weakref, types, copy, pprint, reprlib and enum.

Highlights

  • Don't use datetime.datetime.time(), use timetz().
  • I'll be happy to forget calendar in a few days.
  • All of collections is good and underused.
    • deques have a maxlen attribute and will discard items past that length.
  • bisect maintains sorted lists cheaply.
  • Trigger callbacks on garbage collection with weakref.finalize(obj, callback)
  • You can only weak ref list and dict subclasses, not the original types themselves.
  • types.coroutine() can turn a generator function into an awaitable coroutine.
  • If an enum has two members with the same value, the second one is an alias for the first.

Read more…

December 1: Python Text Processing Services

As part of the Python Standard Library traversal, today, we're going through the Python text processing services: string, re, difflib, textwrap, unicodedata, stringprep, readline, rlcompleter.

Highlights

  • string.Template is kind of nice for user-facing string substitution.
  • string.capwords() runs str.capitalize() on every word in a string.
  • difflib.get_close_matches can help you catch user typos.
  • re.split is like "".split, but for regular expressions.
  • re.sub can take a function instead of a replacement text.
  • People who don't know about textwrap: textwrap is good, use it! Particularly text shortening and indent/dedent functionality is a really nice add-on.

Read more…

November 30: Built-in Constants, Types, Exceptions

After yesterday's built-in functions, today's entry in the traversal of the Python Standard Library docs takes us to the built-in constants, types and exceptions.

Highlights

The things I learned or found most interesting in this post:

  • I keep forgetting that divmod exists, oops.
  • Use string.casefold() for case-insensitive comparison, and expandtab() to replace tabs with spaces, without reverse operation, mwahaha.
  • You can assign to list slices (duh), and that includes stepped slices ([2:10:3]) (waat)
  • I played a bit with memoryview, which lets you access and manipulate bytes and bytearrays directly by exposing the raw integers.
  • I didn't know that if you use my_set.union() you can actually use any iterable as argument (as opposed to my_set | other_set where other_set really needs to be a set)

Read more…

First Sunday of Advent: Built-in Functions

This post is part of my series on Traversing the Python Standard Library.

Built-in functions are included in the standard library and are always available with no need for imports. There are 69 (nice) of them. I've rearranged them from the Python documentation order (alphabetical) into groups – this was short enough to do a detailed breakdown, the following days will be more of a summary.

Highlights

These are the things I did not know or had forgotten:

  • open() takes an errors argument that you can set to surrogateescape, which allows you to process and even roundtrip encoding errors in files.
  • iter() can take two arguments and then calls the first one repeatedly until its result equals the second argument, good for chunked reading.
  • round(), when it could round either way, chooses the even number, so both round(1.5) and round(2.5) is 2.

Read more…

0. Introduction: Traversing the Python Standard Library Documentation

I like Python.

I've been a professional programmer for at least five years now, depending on how you count, and I've spent a majority of that time with Python. I like it. It's comfortable, it's fun, it doesn't get in my way too much most of the time, it has a rich ecosystem filled with good people who make work pleasant, and many of who have turned into friends over the years.

And yet … I have also never gotten over a slightly imposter-syndrome tinged sense of unease around the language. Sure, I've been using it for close to a decade, and sure, I've been putting in time and effort to learn new things and avoid bad patterns. When called upon to relay my experience level, I call myself fluent or experienced or proficient. And yet …

Read more…

Custom exception reporters in Django

If your Django project runs into exceptions in production, Django can email you a complete stacktrace, and all the context information you'd usually see in development mode. As of Django 3.0, you can tweak the content of these emails (and Django 3.1 made it a bit easier for us, so this blog post assumes Django 3.1 or later releases).

Read more…