December 13: Python Concurrent Execution Modules

Now we get to talk about a completely trivial topic, barely worth mentioning: Concurrent execution, including threading, multiprocessing, concurrent, subprocess, sched and queue.

Now, you might say, "wait a second, it is not December 13th!", and … you would be right. Life got in the way and I had to spend a week figuring some things out, and pushing out a blog post about concurrency in Python did not quite fit in with that. Now all the things are figured out, and I'm continuing the Traversal of the Python Standard Library Documentation.

Highlights

  • Oh god Popen has so many arguments.
  • Python ships a "general purpose event scheduler"

Read more…

December 11: Python Operating System Services

After some restful days, today's Python Standard Library Traversal follows up yesterday's Cryptographic Service Modules with all the generic OS interaction modules. That is os (but not os.path, we did that a week ago), io, time, argparse, getopt, logging and its submodules, getpass, curses and submodules, platform, errno and ctypes. Strap in!

Highlights

  • On Windows, os.startfile() acts like double-clicking the file.
  • There are way too many ways to start new processes in Python.
  • "seconds since epoch" usually excludes leap seconds.
  • argparse is honestly not as bad as I remembered it. Stockholm syndrome?
  • logger.getChild() is like calling getLogger() on the full target name, very useful when you use stand-in values like __name__.
  • logger.handlers.TimedRotatingFileHandler and logger.handlers.RotatingFileHandler handle log file rotation, nice!
  • logger.handlers.HTTPHandler defaults to secure=False.
  • You can change the default %-style string formatting in logging formatters to str.format() or string.Template.
  • You can query the platform you are running on with the platform module.

Read more…

December 9: Python File Format Modules

Combining the commonplace and the weirdly specific: As part of the Python Standard Library traversal, after yesterday's short post about data compression and archiving modules, today we're going to look at Python file format modules. Those include csv, configparser, netrc, xdrlib and the forgettable plistlib.

Highlights

  • Writing your own csv dialect is extremely easy. I wonder how cursed you can make it.
  • configparser has an extended interpolation mode that allows you to refer to values from other sections like %{OtherSection:other_value}
  • You can configure lots of things about configparser, like the delimiters, comment prefix, strict mode, enable multiline strings with empty lines, permit empty/flag values, …
  • Why on earth is csv in a different documentation block from json and xml?
  • I learnt that the netrc FTP configuration format exists, and requires its own stdlib support module.
  • I learnt that xdr file format exists, and at least the Python interface sounds pretty annoying.
  • I learnt that plist Apple format exists, and it sounds not that bad.

Read more…

December 8: Pyton Data Compression and Archiving Modules

As part of the Python Standard Library traversal, after yesterday's short post about data persistence, today we're going to look at Python data compression modules. There are not too many of them, but definitely more than I knew: zlib, gzip, bz2, lzma, zipfile, tarfile.

Highlights

  • Now I know that Python supports six different main compression protocols out of the box.
  • The zipfile module has a CLI interface for compression, extraction and listing of archives.
  • Same for tarfile.
  • When opening a tarfile, you can pass the intended compression algorithm with the file mode, like w:bz2.

Read more…

December 7: Python Data Persistence Modules

Strap in, this is a long one: As part of the Python Standard Library traversal, after yesterday's short post about binary data services, today we're going to look at Python data type modules. There's a lot of them: datetime, calendar, collections, heapq, bisect, array, weakref, types, copy, pprint, reprlib and enum.

Highlights

  • You can use shelve to store pickled objects in a persistent key-value store.
  • The Unix dbm library is a thing that exists and has Python standard library support.

Read more…

December 6: Python File and Directory Access

There's no place where "one obvious way to do things" fails as much as it does with file and OS interaction. As part of the Python Standard Library traversal, after yesterday's post about functional programming modules, today we're going to look at the file and directory access modules. Yes, all of them: pathlib, os.path, fileinput, stat, filecmp, tempfile, glob, fnmatch, linecache and shutil.

Highlights

  • The 80% overlap between pathlib and os.path is bordering on hilarious. One obvious way indeed.
  • fileinput is really weird.
  • stat allows you to query and extract results of os.stat()
  • shutil.diskusage() returns total, used and free bytes for a given directory.
  • shutil.which() looks up executables (platform independent which)
  • shutil.make_archive supports zip, tar, gztar, bztar and xtar out of the box.

Read more…

December 5: Python Functional Programming Modules

A short entry for a change: As part of the Python Standard Library traversal, after yesterday's short post about Python's numeric and mathematical modules, I'm taking a look at itertools, functools and operator.

Highlights

  • There really are a lot of itertools functions.
  • functools contains a cached_property decorator.
  • I finally know what lru_cache does.
  • Apparently Python has a type of function overloading, by type of first argument, with singledispatch.
  • There are operator functions for performing in-place operations like +=.
  • operator.attrgetter can retrieve multiple and nested attributes.
  • operator.itemgetter can retrieve multiple items.

Read more…

December 4: Python Numeric and Mathematical Modules

Today is a bit less bad than yesterday: As part of the Python Standard Library traversal, after yesterday's extensive post about data types, today we're going to look at Python modules that take care of numbers and maths and make us very happy because we don't need to implement them ourselves. Or wait for NumPy to install.

Highlights

  • The numbers module contains abstract base classes so you can check for features of given numbers using isinstance.
  • The Python documentation takes a stand for tau and against pi. Good documentation.
  • cmath provides mathematical operations and functions for complex numbers – it exists separately from math so that people aren't surprised when they run math.sqrt(-1). 🤣
  • fractions.limit_denominator() allows you to take a float and turn it into the number it was meant to be. Go team fractions.
  • The fact that decimal is singular and fractions is plural will never cease to confuse me.
  • Apparently there's both random.gauss() and random.normalvariate() ???
  • statistics.NormalDist creates a normal distribution either from a set of samples or the mean and a standard deviation.

Read more…

December 3: Python Data Types

Strap in, this is a long one: As part of the Python Standard Library traversal, after yesterday's short post about binary data services, today we're going to look at Python data type modules. There's a lot of them: datetime, calendar, collections, heapq, bisect, array, weakref, types, copy, pprint, reprlib and enum.

Highlights

  • Don't use datetime.datetime.time(), use timetz().
  • I'll be happy to forget calendar in a few days.
  • All of collections is good and underused.
    • deques have a maxlen attribute and will discard items past that length.
  • bisect maintains sorted lists cheaply.
  • Trigger callbacks on garbage collection with weakref.finalize(obj, callback)
  • You can only weak ref list and dict subclasses, not the original types themselves.
  • types.coroutine() can turn a generator function into an awaitable coroutine.
  • If an enum has two members with the same value, the second one is an alias for the first.

Read more…

December 1: Python Text Processing Services

As part of the Python Standard Library traversal, today, we're going through the Python text processing services: string, re, difflib, textwrap, unicodedata, stringprep, readline, rlcompleter.

Highlights

  • string.Template is kind of nice for user-facing string substitution.
  • string.capwords() runs str.capitalize() on every word in a string.
  • difflib.get_close_matches can help you catch user typos.
  • re.split is like "".split, but for regular expressions.
  • re.sub can take a function instead of a replacement text.
  • People who don't know about textwrap: textwrap is good, use it! Particularly text shortening and indent/dedent functionality is a really nice add-on.

Read more…

November 30: Built-in Constants, Types, Exceptions

After yesterday's built-in functions, today's entry in the traversal of the Python Standard Library docs takes us to the built-in constants, types and exceptions.

Highlights

The things I learned or found most interesting in this post:

  • I keep forgetting that divmod exists, oops.
  • Use string.casefold() for case-insensitive comparison, and expandtab() to replace tabs with spaces, without reverse operation, mwahaha.
  • You can assign to list slices (duh), and that includes stepped slices ([2:10:3]) (waat)
  • I played a bit with memoryview, which lets you access and manipulate bytes and bytearrays directly by exposing the raw integers.
  • I didn't know that if you use my_set.union() you can actually use any iterable as argument (as opposed to my_set | other_set where other_set really needs to be a set)

Read more…

First Sunday of Advent: Built-in Functions

This post is part of my series on Traversing the Python Standard Library.

Built-in functions are included in the standard library and are always available with no need for imports. There are 69 (nice) of them. I've rearranged them from the Python documentation order (alphabetical) into groups – this was short enough to do a detailed breakdown, the following days will be more of a summary.

Highlights

These are the things I did not know or had forgotten:

  • open() takes an errors argument that you can set to surrogateescape, which allows you to process and even roundtrip encoding errors in files.
  • iter() can take two arguments and then calls the first one repeatedly until its result equals the second argument, good for chunked reading.
  • round(), when it could round either way, chooses the even number, so both round(1.5) and round(2.5) is 2.

Read more…

0. Introduction: Traversing the Python Standard Library Documentation

I like Python.

I've been a professional programmer for at least five years now, depending on how you count, and I've spent a majority of that time with Python. I like it. It's comfortable, it's fun, it doesn't get in my way too much most of the time, it has a rich ecosystem filled with good people who make work pleasant, and many of who have turned into friends over the years.

And yet … I have also never gotten over a slightly imposter-syndrome tinged sense of unease around the language. Sure, I've been using it for close to a decade, and sure, I've been putting in time and effort to learn new things and avoid bad patterns. When called upon to relay my experience level, I call myself fluent or experienced or proficient. And yet …

Read more…

Custom exception reporters in Django

If your Django project runs into exceptions in production, Django can email you a complete stacktrace, and all the context information you'd usually see in development mode. As of Django 3.0, you can tweak the content of these emails (and Django 3.1 made it a bit easier for us, so this blog post assumes Django 3.1 or later releases).

Read more…

Static websites from data files

Sometimes, you want to publish structured data in a way that is nice for people to look at. Maybe you have a CSV file with the export from a tool you sometimes use, or the JSON data from somebody's API. Instead of using the data in a program, you'd like the data to be readable for everybody, and maybe even nice to look at? With a way to provide seamless updates, even?

I recently found a nice solution to this problem using Jekyll, GitHub (Pull Requests, Pages, and Actions).

Read more…

∞ Tannenbaum!

Source.

On the first Sunday of Advent, my siblings (13 & 15) and I got busy building Christmas trees – but not out of wood! We made generative art. You can see the result at 🎄.rixx.de (fallback for the less unicode-savvy: ootannenbaum.rixx.de), where you can generate your own random Christmas trees.

That's already pretty cool, in my admittedly biased opinion. What's even cooler though is how we got there: While I did all the typing, everything else was a collaboration. All the research, ideas, testing, calculations that went into this, roughly four hours of joint generative Christmas. What we did can work for others, too, so this is a writeup of what we did, how this works, and how to tackle generative processes like these. It's much easier than it may look if you haven't done something like this before!

Resources

First of all: You can look up all the details of how the Christmas tree is made. This file contains all the code, and it's heavily commentated (close to be its own blog post). The comments walk through the code and explain in plain English what's going on. If you leave out the ".en" in the URL, you'll find the same in German (which is the version we actually wrote).

If you haven't done anything remotely similar before, there is a great introduction in German by the equally great bleeptrack. This is a similar talk in English. You'll find more links and resources at the end of this page.

Prior knowledge

What level of prior knowledge do you need to build something like this? Don't worry, it's not all that much! A bit of perseverance or knowledge of a bit of any programming language or somebody who can program and is willing to help you should show you through. If you don't know anybody who can help you, look for local hackerspaces, programming courses, or ask around online when you hit stumbling blocks. Given concrete questions, there should be people around who can help.

For reference: my brother is 15, has learned some Java (and a tiny bit of HTML) at school, and had no problems following our progress. My sister is 13, and has no programming experience at all. She still understood what we did as well, especially the design and work on the parameters to get to a balanced result.

Process

When starting out with a generator like this, it's best to start simple: Think of the basic form hidden in what you're trying to do, and try to get it to show up in the live editor at sketch.paperjs.org. For example: to draw a rectangle, you can copy the code below and click the “run” button at the top:

var rectangle = new Path.Rectangle(new Point(100, 100), new Size(200, 300));
rectangle.fillColor = "red";
rectangle.strokeColor = "black";
rectangle.strokeWidth = 5;
rectangle.rotate(45)

As somebody who speaks English, you have an advantage, because this is pretty self-explanatory for you. It draws a rectangle at the coordinate (100, 100). Note that (0, 0) is the top left corner. The first number is the x value, which changes how far something is moved to the right, and the second number is the y value, which moves elements further down.

The rectangle takes up 200x300 pixels, and filled with red colour with a 5 pixel black border. Finally, we rotate it by 45°. Now try to play around with it a bit! You can read up on all the other possibilities in the documentation. Don't worry if it's hard to read at first, that's completely normal when approaching technical documentation. It's best to get started by reading the documentation for some part that you know already, to figure out the structure and language. For example, search for “rotate” to see the description of the rotation of an object.

For our Christmas tree, we started out by drawing triangles on top of each other. So we had to figure out the triangle coordinates, so that the point on top was centered between the other two. We started out on paper to make sure we understood the coordinate system, and then transfered our result to the paper.js editor. Then we played around with the numbers until we could draw first one, then several triangles.

Advice

The most important piece of advice is: proceed in very small steps. After each change, take a look at the results and if they match your expectations. The nice thing about a graphical project like this is that you can see if your code works as expected, and if it doesn't, the results are often very funny. More than once we generated trees that were unreasonably tiny, or huge, entirely transparent, or upside down!

Small steps could be: Draw one element. Draw another element. Add colour. Add a third element in a different colour. Change the width of the element. Make the width random. Make the amount of elements random. And so on.

The second most important piece of advice is: learn from others. The vaaaaaaaast majority of problems in this area have been encountered and presumably solved by others. It's good to try on your own at first – that way you learn more about the problem and its complexities. But if you can't find a solution soon, instead of growing frustrated: Search (e.g. on GitHub) for people with similar code and hopefully solutions. We did this a couple of times – for instance, the code to export the generated trees to PNG and SVG images is taken directly from a similar generator by bleeptrack. And now you can look up similar things in turn in our extensively commented code.

Randomness

After some time and experimenting, you'll end up with a drawing. Now you can add in the part that's the most fun: Randomness! It's not terribly easy, but a lot of fun, and will reward you with unexpected results.

JavaScript has a function named Math.random(), which returns a long number between 0 and 1. We built a small helper function which returns a random number between its parameters:

function randomNumber(min, max) {
  // This function produces a random number between min and max.
  // Min is inclusive, max is exclusive.
  return Math.floor(Math.random() * (max - min)) + min;
}

Feel free to copy this function, because building on this it's way easier to get started. Now you can do things like make your drawing between 100 and 300 pixels wide, or make it rotate by 45°-90°. You can also use this like dice or throwing a coin to choose between different features. We only draw a star on top of our Christmas tree for two out of three trees, by calling randomNumber(0, 3), and only drawing a star if the result is not 0.

Colour

The size of a drawing is the best first approach to randomness. It will make you think about some measurements some more: If parts of your drawing border a random part, they need to be positioned correctly, so you'll have to add some calculations for that. If this gets boring, you can also vary the colour.

Colours in paper.js and in many other digital systems are made up of three parts: red, green, and blue (the RGB system). These values are often on a scale from 0 to 255. paper.js uses values between 0 and 1 instead – but because I found it harder to reason about small numbers like that, we chose numbers in the [0, 255] range and divided them by 256, instead. This is how we determine the green colour of a tree:

var red = randomNumber(0, 100) / 256
var green = randomNumber(115, 255) / 256
var blue = randomNumber(40, 130) / 256
triangle.fillColor = new Color(red, green, blue)

So how do we find these colour ranges? How do we know that our red value should be between 0 and 100? If you search for Colourpicker in the search engine of your choice, you should find an interactive colourpicker, often as part of the search result page. If you move the colour point in the area that looks good to you, you can pay attention to the way the three values change, and which range is appropriate for each of them. This is way easier if you're not alone, by the way, because one of you can focus on moving the point in a good colour space, and the other can write down the values. Afterwards, you can finetune the colours, e.g. if you notice that your colours tend to come out too blue, you can reduce the upper limit of your blue range.

Further reading

And this is how you can get started with generative art! I hope this helps to get you (or people you know) started. Have fun!

This is a list of sources and further reading: