It's been a while since I posted a huge book lists. I try to contain all my book posting impulses over at books.rixx.de these days, after all. (Which also has an RSS feed, for those of you following this blog and concerned about the silence). However, I took a dangerous step this year and became part of a Sci-Fi book club – a very ill-advised move, as we all fuel each other's addiction. Thankfully, my addiction was already well-fuelled, so it's not like the book club is doing anything but slightly accellerate my descent into madness. It's fine, it's fine!

Apart from the occasional nearly violent civil debate about the boundaries of Sci-Fi and Fantasy, we're a peaceful lot, I thought. That is, until one of us prompted the group to name our top three Sci-Fi books. Three!, dear reader, not three hundred, or at least thirty! And while grieviously wounded – and coping by posting increasingly long lists of Sci-Fi books, to reassure each other that, yes, we weren't about to run out of books, M. said:

Your task, should you choose to accept it, is to craft a list of 41 Fantasy books, each of which must not only be essential reading but also be the soulmate of a Discworld novel (only one book per DW-novel - if we start accepting poligamy into our Fantasy reading list there’s no saying where we’ll end up). Go!

And here we are.

Read more…

A year, gone, and for what? Let's see. Wrote some code, read some books, made some friends, and occasionally even had a life. A year in review:

Read more…

Hi, Future Me! It seems that you've forgotten once again how Django's @page_cache decorator function works. But this time, I'm here to tell you what you (or rather, I) worked out in the past. So here it is: @page_cache from the ground up.

Read more…

Together with a friend, I’m running the Mastodon/Fediverse instance chaos.social. We’ve been at it for nearly five years now, and I feel like it’s time to reflect on the good, the bad, the ugly. Please note that you'll be reading an edited version of this post with most of the cursing removed, so please be understanding of the bits that I kept – they are load-bearing.

Read more…

This is an edited transcript of the keynote I delivered at PyCon UK in 2019. I'm still grateful to the organisers, and I still treasure my memories of visiting Cardiff for the first time that year. I've edited the transcript only to make it readable, fixing sentences and removing filler words – but the content and structure remains unchanged. I also left in audience reactions, for flair (and because I'm a bit proud to have made people laugh). See the end notes for corrections and other remarks – you can also see the full slides including speaker notes here.

Read more…

Now we get to talk about a completely trivial topic, barely worth mentioning: Concurrent execution, including threading, multiprocessing, concurrent, subprocess, sched and queue.

Now, you might say, "wait a second, it is not December 13th!", and … you would be right. Life got in the way and I had to spend a week figuring some things out, and pushing out a blog post about concurrency in Python did not quite fit in with that. Now all the things are figured out, and I'm continuing the Traversal of the Python Standard Library Documentation.

Highlights

• Oh god Popen has so many arguments.
• Python ships a "general purpose event scheduler"

Read more…

As part of the Python Standard Library documentation traversal, we're following up on yesterday's exhausting post about Operating System Services with probably the shortest chapter yet: the contextvars module. It's not cheating, it really is listed as its own high-level documentation section!

Read more…

After some restful days, today's Python Standard Library Traversal follows up yesterday's Cryptographic Service Modules with all the generic OS interaction modules. That is os (but not os.path, we did that a week ago), io, time, argparse, getopt, logging and its submodules, getpass, curses and submodules, platform, errno and ctypes. Strap in!

Highlights

• On Windows, os.startfile() acts like double-clicking the file.
• There are way too many ways to start new processes in Python.
• "seconds since epoch" usually excludes leap seconds.
• argparse is honestly not as bad as I remembered it. Stockholm syndrome?
• logger.getChild() is like calling getLogger() on the full target name, very useful when you use stand-in values like __name__.
• logger.handlers.TimedRotatingFileHandler and logger.handlers.RotatingFileHandler handle log file rotation, nice!
• logger.handlers.HTTPHandler defaults to secure=False.
• You can change the default %-style string formatting in logging formatters to str.format() or string.Template.
• You can query the platform you are running on with the platform module.

Read more…

Shhh! 🤫 As part of the Python Standard Library traversal, after yesterday's look at file format modules, today is going to be all about secrets. Well, all about secrets and hmac and hashlib, to be precise.

Highlights

• There's an awful lot of hash algorithms in hashlib.
• The secrets module exists and is suitable for (reasonably) strong randomness, eg for generating long secret URLs.

Read more…

Combining the commonplace and the weirdly specific: As part of the Python Standard Library traversal, after yesterday's short post about data compression and archiving modules, today we're going to look at Python file format modules. Those include csv, configparser, netrc, xdrlib and the forgettable plistlib.

Highlights

• Writing your own csv dialect is extremely easy. I wonder how cursed you can make it.
• configparser has an extended interpolation mode that allows you to refer to values from other sections like %{OtherSection:other_value}
• You can configure lots of things about configparser, like the delimiters, comment prefix, strict mode, enable multiline strings with empty lines, permit empty/flag values, …
• Why on earth is csv in a different documentation block from json and xml?
• I learnt that the netrc FTP configuration format exists, and requires its own stdlib support module.
• I learnt that xdr file format exists, and at least the Python interface sounds pretty annoying.
• I learnt that plist Apple format exists, and it sounds not that bad.

Read more…

As part of the Python Standard Library traversal, after yesterday's short post about data persistence, today we're going to look at Python data compression modules. There are not too many of them, but definitely more than I knew: zlib, gzip, bz2, lzma, zipfile, tarfile.

Highlights

• Now I know that Python supports six different main compression protocols out of the box.
• The zipfile module has a CLI interface for compression, extraction and listing of archives.
• Same for tarfile.
• When opening a tarfile, you can pass the intended compression algorithm with the file mode, like w:bz2.

Read more…

Strap in, this is a long one: As part of the Python Standard Library traversal, after yesterday's short post about binary data services, today we're going to look at Python data type modules. There's a lot of them: datetime, calendar, collections, heapq, bisect, array, weakref, types, copy, pprint, reprlib and enum.

Highlights

• You can use shelve to store pickled objects in a persistent key-value store.
• The Unix dbm library is a thing that exists and has Python standard library support.

Read more…

There's no place where "one obvious way to do things" fails as much as it does with file and OS interaction. As part of the Python Standard Library traversal, after yesterday's post about functional programming modules, today we're going to look at the file and directory access modules. Yes, all of them: pathlib, os.path, fileinput, stat, filecmp, tempfile, glob, fnmatch, linecache and shutil.

Highlights

• The 80% overlap between pathlib and os.path is bordering on hilarious. One obvious way indeed.
• fileinput is really weird.
• stat allows you to query and extract results of os.stat()
• shutil.diskusage() returns total, used and free bytes for a given directory.
• shutil.which() looks up executables (platform independent which)
• shutil.make_archive supports zip, tar, gztar, bztar and xtar out of the box.

Read more…

A short entry for a change: As part of the Python Standard Library traversal, after yesterday's short post about Python's numeric and mathematical modules, I'm taking a look at itertools, functools and operator.

Highlights

• There really are a lot of itertools functions.
• functools contains a cached_property decorator.
• I finally know what lru_cache does.
• Apparently Python has a type of function overloading, by type of first argument, with singledispatch.
• There are operator functions for performing in-place operations like +=.
• operator.attrgetter can retrieve multiple and nested attributes.
• operator.itemgetter can retrieve multiple items.

Read more…

Today is a bit less bad than yesterday: As part of the Python Standard Library traversal, after yesterday's extensive post about data types, today we're going to look at Python modules that take care of numbers and maths and make us very happy because we don't need to implement them ourselves. Or wait for NumPy to install.

Highlights

• The numbers module contains abstract base classes so you can check for features of given numbers using isinstance.
• The Python documentation takes a stand for tau and against pi. Good documentation.
• cmath provides mathematical operations and functions for complex numbers – it exists separately from math so that people aren't surprised when they run math.sqrt(-1). 🤣
• fractions.limit_denominator() allows you to take a float and turn it into the number it was meant to be. Go team fractions.
• The fact that decimal is singular and fractions is plural will never cease to confuse me.
• Apparently there's both random.gauss() and random.normalvariate() ???
• statistics.NormalDist creates a normal distribution either from a set of samples or the mean and a standard deviation.

Read more…

Strap in, this is a long one: As part of the Python Standard Library traversal, after yesterday's short post about binary data services, today we're going to look at Python data type modules. There's a lot of them: datetime, calendar, collections, heapq, bisect, array, weakref, types, copy, pprint, reprlib and enum.

Highlights

• Don't use datetime.datetime.time(), use timetz().
• I'll be happy to forget calendar in a few days.
• All of collections is good and underused.
  • deques have a maxlen attribute and will discard items past that length.
• bisect maintains sorted lists cheaply.
• Trigger callbacks on garbage collection with weakref.finalize(obj, callback)
• You can only weak ref list and dict subclasses, not the original types themselves.
• types.coroutine() can turn a generator function into an awaitable coroutine.
• If an enum has two members with the same value, the second one is an alias for the first.

Read more…

As part of the Python Standard Library traversal, today we're following up yesterday's text processing services with the binary data services: struct and codecs.

Highlights

• codecs.encode("foo", "rot13") 😍
• Now I know about yet another formatting language in Python. Thanks, struct.

Read more…

As part of the Python Standard Library traversal, today, we're going through the Python text processing services: string, re, difflib, textwrap, unicodedata, stringprep, readline, rlcompleter.

Highlights

• string.Template is kind of nice for user-facing string substitution.
• string.capwords() runs str.capitalize() on every word in a string.
• difflib.get_close_matches can help you catch user typos.
• re.split is like "".split, but for regular expressions.
• re.sub can take a function instead of a replacement text.
• People who don't know about textwrap: textwrap is good, use it! Particularly text shortening and indent/dedent functionality is a really nice add-on.

Read more…

After yesterday's built-in functions, today's entry in the traversal of the Python Standard Library docs takes us to the built-in constants, types and exceptions.

Highlights

The things I learned or found most interesting in this post:

• I keep forgetting that divmod exists, oops.
• Use string.casefold() for case-insensitive comparison, and expandtab() to replace tabs with spaces, without reverse operation, mwahaha.
• You can assign to list slices (duh), and that includes stepped slices ([2:10:3]) (waat)
• I played a bit with memoryview, which lets you access and manipulate bytes and bytearrays directly by exposing the raw integers.
• I didn't know that if you use my_set.union() you can actually use any iterable as argument (as opposed to my_set | other_set where other_set really needs to be a set)

Read more…

This post is part of my series on Traversing the Python Standard Library.

Built-in functions are included in the standard library and are always available with no need for imports. There are 69 (nice) of them. I've rearranged them from the Python documentation order (alphabetical) into groups – this was short enough to do a detailed breakdown, the following days will be more of a summary.

Highlights

These are the things I did not know or had forgotten:

• open() takes an errors argument that you can set to surrogateescape, which allows you to process and even roundtrip encoding errors in files.
• iter() can take two arguments and then calls the first one repeatedly until its result equals the second argument, good for chunked reading.
• round(), when it could round either way, chooses the even number, so both round(1.5) and round(2.5) is 2.

Read more…