December 5: Python Functional Programming Modules

A short entry for a change: As part of the Python Standard Library traversal, after yesterday's short post about Python's numeric and mathematical modules, I'm taking a look at itertools, functools and operator.

Highlights

• There really are a lot of itertools functions.
• functools contains a cached_property decorator.
• I finally know what lru_cache does.
• Apparently Python has a type of function overloading, by type of first argument, with singledispatch.
• There are operator functions for performing in-place operations like +=.
• operator.attrgetter can retrieve multiple and nested attributes.
• operator.itemgetter can retrieve multiple items.

itertools

Do you like your iterators to be efficient and fast? itertools has your back then.

infinite

Some iterators are infinite: count() starts at an integer and just counts up, optionally by a step. cycle() cycles through a given sequence forever. repeat() repeats an item endlessly or n times.

finite

Among the terminating functions: accumulate() increasingly adds up the elements in the iterable. chain() collapses several iterables into a single one. compress() takes two iterables and returns only elements of the first iterable when the same index in the second iterable is True-ish. takewhile() gives you all of a sequence until the predicate fails and dropwhile() returns all of the sequence starting from the item where the predicate fails. filterfalse() inverts the filter function. groupby() groups an iterable by a given key or keyfunc. islice() gives you elements from an iterable, starting at start, optionally to stop, via step. tee() turns one iterable into n iterables. zip_longest works until the longest iterable is done and fills the others with a given filler value. starmap() is like map(), except that it passes an unpacked set to the function being mapped to.

combinatoric

Finally, there are combinatoric functions. product() combines all items of all iterables with one another. permutations() returns tuples of a given length, with all possible orderings and no repeated elements. combinations() returns tuples of a given length, already ordered with no repeated elements. combinations_with_replacements() includes repeated elements.

functools

functools contains functions that act on other functions. All the way down.

reduce() applies the given function cumulatively to the items of an iterable to reduce it to a single value.

singledispatch and singledispatchmethod give you a way to create overloaded functions, based on the type of the first argument to the function.

@wraps is a convenience function to create proper wrapping functions for decorators, adding the correct docstrings and so on.

caching

@cached_property creates a property that is evaluated only once, and on repeated access, the cached result is returned. Extremely useful for heavy computation in properties.

Similarly, @lru_cache wraps a function and retains results up to a memory limit. You can access cache information with cache_info(), returning hits, misses and maxsize. You can bypass the cache via __wrapped__ and clear it with cache_clear(). Use @cache if you want to use lru_cache with no maxsize (new in 3.9).

ordering

cmp_to_key takes a regular function useful for comparisons and transforms it into a function usable by key= interfaces like in sorted(). You use it like sorted(i, key=cmp_to_key(locale.strcoll)).

@total_ordering is a class decorator, that given a class that defines __eq__ and one comparison method, will add all the other comparison functions.

partials

partial takes a function and arguments, and returns a function (technically, a partial object) that will always be called with those arguments, plus additional ones, for example basetwo = partial(int, base=2). You can also use partialmethod() to do the same on classes.

operator

The operator module exposes functions corresponding to intrinsic Python operators, like operator.add() duplicating a+b behaviour. There is also a set of in-place operators, replacing operations like += – all of those just have an initial i, like iadd().

The probably most used functions are the attribute lookup functions – they are very useful as field extractors for functions like sorted() and groupby(). They include attrgetter (for nested and multiple attributes!), itemgetter and methodcaller.

The comparison functions are lt, le, eq, ne, ge, gt, not, truth, is_, is_not. The mathematical and logical functions are abs, add, and_, floordiv, index, invert, lshift, mod, mul, matmul, neg, or_, pos, pow, rshift, sub, truediv, xor. Functions for sequences are concat, contains, countOf and indexOf, getitem, setitem and delitem, plus length_hint.