How does the functools cmp_to_key function work?
In Python, both list.sort method and sorted built-in function accepts an optional parameter named key , which is a function that, given an element from the list returns its sorting key. Older Python versions used a different approach using the cmp parameter instead, which is a function that, given two elements from the list returns a negative number if the first is less than the second, zero if there are equals and a positive number if the first is greater. At some point, this parameter was deprecated and wasn’t included in Python 3. The other day I wanted to sort a list of elements in a way that a cmp function was much more easier to write than a key one. I didn’t wanted to use a deprecated feature so I read the documentation and I found that there is a funtion named cmp_to_key in the functools module which, as his name states, receives a cmp function and returns a key one. or that’s what I thought until I read the source code (or at least an equivalent version) of this high level function included in the docs
def cmp_to_key(mycmp): 'Convert a cmp= function into a key= function' class K(object): def __init__(self, obj, *args): self.obj = obj def __lt__(self, other): return mycmp(self.obj, other.obj) < 0 def __gt__(self, other): return mycmp(self.obj, other.obj) >0 def __eq__(self, other): return mycmp(self.obj, other.obj) == 0 def __le__(self, other): return mycmp(self.obj, other.obj) = 0 def __ne__(self, other): return mycmp(self.obj, other.obj) != 0 return K Despite the fact that cmp_to_key works as expected, I get surprised by the fact that this function doesn’t return a function but a K class instead. Why? How does it work? My guess it that the sorted function internally checks whether cmp is a function or a K class or something similar, but I’m not sure. P.S.: Despite this weirdness, I found that K class is very useful. Check this code:
from functools import cmp_to_key def my_cmp(a, b): # some sorting comparison which is hard to express using a key function class MyClass(cmp_to_key(my_cmp)): . This way, any list of instances of MyClass can be, by default, sorted by the criteria defined in my_cmp
Sorting HOW TO¶
Python lists have a built-in list.sort() method that modifies the list in-place. There is also a sorted() built-in function that builds a new sorted list from an iterable.
In this document, we explore the various techniques for sorting data using Python.
Sorting Basics¶
A simple ascending sort is very easy: just call the sorted() function. It returns a new sorted list:
>>> sorted([5, 2, 3, 1, 4]) [1, 2, 3, 4, 5]
You can also use the list.sort() method. It modifies the list in-place (and returns None to avoid confusion). Usually it’s less convenient than sorted() — but if you don’t need the original list, it’s slightly more efficient.
>>> a = [5, 2, 3, 1, 4] >>> a.sort() >>> a [1, 2, 3, 4, 5]
Another difference is that the list.sort() method is only defined for lists. In contrast, the sorted() function accepts any iterable.
>>> sorted(1: 'D', 2: 'B', 3: 'B', 4: 'E', 5: 'A'>) [1, 2, 3, 4, 5]
Key Functions¶
Both list.sort() and sorted() have a key parameter to specify a function (or other callable) to be called on each list element prior to making comparisons.
For example, here’s a case-insensitive string comparison:
>>> sorted("This is a test string from Andrew".split(), key=str.lower) ['a', 'Andrew', 'from', 'is', 'string', 'test', 'This']
The value of the key parameter should be a function (or other callable) that takes a single argument and returns a key to use for sorting purposes. This technique is fast because the key function is called exactly once for each input record.
A common pattern is to sort complex objects using some of the object’s indices as keys. For example:
>>> student_tuples = [ . ('john', 'A', 15), . ('jane', 'B', 12), . ('dave', 'B', 10), . ] >>> sorted(student_tuples, key=lambda student: student[2]) # sort by age [('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]
The same technique works for objects with named attributes. For example:
>>> class Student: . def __init__(self, name, grade, age): . self.name = name . self.grade = grade . self.age = age . def __repr__(self): . return repr((self.name, self.grade, self.age)) >>> student_objects = [ . Student('john', 'A', 15), . Student('jane', 'B', 12), . Student('dave', 'B', 10), . ] >>> sorted(student_objects, key=lambda student: student.age) # sort by age [('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]
Operator Module Functions¶
The key-function patterns shown above are very common, so Python provides convenience functions to make accessor functions easier and faster. The operator module has itemgetter() , attrgetter() , and a methodcaller() function.
Using those functions, the above examples become simpler and faster:
>>> from operator import itemgetter, attrgetter >>> sorted(student_tuples, key=itemgetter(2)) [('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)] >>> sorted(student_objects, key=attrgetter('age')) [('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]
The operator module functions allow multiple levels of sorting. For example, to sort by grade then by age:
>>> sorted(student_tuples, key=itemgetter(1,2)) [('john', 'A', 15), ('dave', 'B', 10), ('jane', 'B', 12)] >>> sorted(student_objects, key=attrgetter('grade', 'age')) [('john', 'A', 15), ('dave', 'B', 10), ('jane', 'B', 12)]
Ascending and Descending¶
Both list.sort() and sorted() accept a reverse parameter with a boolean value. This is used to flag descending sorts. For example, to get the student data in reverse age order:
>>> sorted(student_tuples, key=itemgetter(2), reverse=True) [('john', 'A', 15), ('jane', 'B', 12), ('dave', 'B', 10)] >>> sorted(student_objects, key=attrgetter('age'), reverse=True) [('john', 'A', 15), ('jane', 'B', 12), ('dave', 'B', 10)]
Sort Stability and Complex Sorts¶
Sorts are guaranteed to be stable. That means that when multiple records have the same key, their original order is preserved.
>>> data = [('red', 1), ('blue', 1), ('red', 2), ('blue', 2)] >>> sorted(data, key=itemgetter(0)) [('blue', 1), ('blue', 2), ('red', 1), ('red', 2)]
Notice how the two records for blue retain their original order so that (‘blue’, 1) is guaranteed to precede (‘blue’, 2) .
This wonderful property lets you build complex sorts in a series of sorting steps. For example, to sort the student data by descending grade and then ascending age, do the age sort first and then sort again using grade:
>>> s = sorted(student_objects, key=attrgetter('age')) # sort on secondary key >>> sorted(s, key=attrgetter('grade'), reverse=True) # now sort on primary key, descending [('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]
This can be abstracted out into a wrapper function that can take a list and tuples of field and order to sort them on multiple passes.
>>> def multisort(xs, specs): . for key, reverse in reversed(specs): . xs.sort(key=attrgetter(key), reverse=reverse) . return xs >>> multisort(list(student_objects), (('grade', True), ('age', False))) [('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]
The Timsort algorithm used in Python does multiple sorts efficiently because it can take advantage of any ordering already present in a dataset.
Decorate-Sort-Undecorate¶
This idiom is called Decorate-Sort-Undecorate after its three steps:
- First, the initial list is decorated with new values that control the sort order.
- Second, the decorated list is sorted.
- Finally, the decorations are removed, creating a list that contains only the initial values in the new order.
For example, to sort the student data by grade using the DSU approach:
>>> decorated = [(student.grade, i, student) for i, student in enumerate(student_objects)] >>> decorated.sort() >>> [student for grade, i, student in decorated] # undecorate [('john', 'A', 15), ('jane', 'B', 12), ('dave', 'B', 10)]
This idiom works because tuples are compared lexicographically; the first items are compared; if they are the same then the second items are compared, and so on.
It is not strictly necessary in all cases to include the index i in the decorated list, but including it gives two benefits:
- The sort is stable – if two items have the same key, their order will be preserved in the sorted list.
- The original items do not have to be comparable because the ordering of the decorated tuples will be determined by at most the first two items. So for example the original list could contain complex numbers which cannot be sorted directly.
Another name for this idiom is Schwartzian transform, after Randal L. Schwartz, who popularized it among Perl programmers.
Now that Python sorting provides key-functions, this technique is not often needed.
Comparison Functions¶
Unlike key functions that return an absolute value for sorting, a comparison function computes the relative ordering for two inputs.
For example, a balance scale compares two samples giving a relative ordering: lighter, equal, or heavier. Likewise, a comparison function such as cmp(a, b) will return a negative value for less-than, zero if the inputs are equal, or a positive value for greater-than.
It is common to encounter comparison functions when translating algorithms from other languages. Also, some libraries provide comparison functions as part of their API. For example, locale.strcoll() is a comparison function.
To accommodate those situations, Python provides functools.cmp_to_key to wrap the comparison function to make it usable as a key function:
sorted(words, key=cmp_to_key(strcoll)) # locale-aware sort order
Odds and Ends¶
- For locale aware sorting, use locale.strxfrm() for a key function or locale.strcoll() for a comparison function. This is necessary because “alphabetical” sort orderings can vary across cultures even if the underlying alphabet is the same.
- The reverse parameter still maintains sort stability (so that records with equal keys retain the original order). Interestingly, that effect can be simulated without the parameter by using the builtin reversed() function twice:
>>> data = [('red', 1), ('blue', 1), ('red', 2), ('blue', 2)] >>> standard_way = sorted(data, key=itemgetter(0), reverse=True) >>> double_reversed = list(reversed(sorted(reversed(data), key=itemgetter(0)))) >>> assert standard_way == double_reversed >>> standard_way [('red', 1), ('red', 2), ('blue', 1), ('blue', 2)]
>>> Student.__lt__ = lambda self, other: self.age other.age >>> sorted(student_objects) [('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]
>>> students = ['dave', 'john', 'jane'] >>> newgrades = 'john': 'F', 'jane':'A', 'dave': 'C'> >>> sorted(students, key=newgrades.__getitem__) ['jane', 'dave', 'john']
How does Python’s cmp_to_key function work?
I came across this function here. I am baffled as to how this would be implemented — how does the key function generated by cmp_to_key know what «position» a given element should be without checking how the given element compares with every other element of interest?
1 Answer 1
The cmp_to_key method returns a special object that acts as a surrogate key:
class K(object): __slots__ = ['obj'] def __init__(self, obj, *args): self.obj = obj def __lt__(self, other): return mycmp(self.obj, other.obj) < 0 def __gt__(self, other): return mycmp(self.obj, other.obj) >0 def __eq__(self, other): return mycmp(self.obj, other.obj) == 0 def __le__(self, other): return mycmp(self.obj, other.obj) = 0 def __ne__(self, other): return mycmp(self.obj, other.obj) != 0 def __hash__(self): raise TypeError('hash not implemented') When sorting, each key will get compared to most other keys in the sequence. Is this element at position 0 lower than or greater than that other object?
Whenever that happens, the special method hooks are invoked, so __lt__ or __gt__ is called, which the surrogate key turns into a call to the cmp method instead.
So the list [1, 2, 3] is sorted as [K(1), K(2), K(3)] , and if, say, K(1) is compared with K(2) to see if K(1) is lower, then K(1).__lt__(K(2)) is called, which is translated to mycmp(1, 2) < 0 .
This is how the old cmp method was working anyway; return -1, 0 or 1 depending on wether the first argument is lower than, equal to or greater than the second argument. The surrogate key translates those numbers back to boolean values for the comparison operators.
At no point does the surrogate key need to know anything about absolute positions. It only needs to know about one other object it is being compared with, and the special method hooks provide that other object.