Python reduce memory consumption

Python consumes a lot of memory or how to reduce the size of objects?

A memory problem may arise when a large number of objects are active in RAM during the execution of a program, especially if there are restrictions on the total amount of available memory.

Below is an overview of some methods of reducing the size of objects, which can significantly reduce the amount of RAM needed for programs in pure Python.

Note: This is english version of my original post (in russian).

For simplicity, we will consider structures in Python to represent points with the coordinates x , y , z with access to the coordinate values by name.

Dict

In small programs, especially in scripts, it is quite simple and convenient to use the built-in dict to represent structural information:

With the advent of a more compact implementation in Python 3.6 with an ordered set of keys, dict has become even more attractive. However, let’s look at the size of its footprint in RAM:

It takes a lot of memory, especially if you suddenly need to create a large number of instances:

Class instance

For those who like to clothe everything in classes, it is preferable to define structures as a class with access by attribute name:

class Point: # def __init__(self, x, y, z): self.x = x self.y = y self.z = z >>> ob = Point(1,2,3) >>> x = ob.x >>> ob.y = y

The structure of the class instance is interesting:

Here __weakref__ is a reference to the list of so-called weak references to this object, the field __dict__ is a reference to the class instance dictionary, which contains the values of instance attributes (note that 64-bit references platform occupy 8 bytes). Starting in Python 3.3, the shared space is used to store keys in the dictionary for all instances of the class. This reduces the size of the instance trace in RAM:

>>> print(sys.getsizeof(ob), sys.getsizeof(ob.__dict__)) 56 112

As a result, a large number of class instances have a smaller footprint in memory than a regular dictionary ( dict ):

It is easy to see that the size of the instance in RAM is still large due to the size of the dictionary of the instance.

Instance of class with __slots__

A significant reduction in the size of a class instance in RAM is achieved by eliminating __dict__ and __weakref__ . This is possible with the help of a «trick» with __slots__ :

class Point: __slots__ = 'x', 'y', 'z' def __init__(self, x, y, z): self.x = x self.y = y self.z = z >>> ob = Point(1,2,3) >>> print(sys.getsizeof(ob)) 64

The object size in RAM has become significantly smaller:

Using __slots__ in the class definition causes the footprint of a large number of instances in memory to be significantly reduced:

Currently, this is the main method of substantially reducing the memory footprint of an instance of a class in RAM.

This reduction is achieved by the fact that in the memory after the title of the object, object references are stored — the attribute values, and access to them is carried out using special descriptors that are in the class dictionary:

>>> pprint(Point.__dict__) mappingproxy( . 'x': , 'y': , 'z': >)

To automate the process of creating a class with __slots__ , there is a library [namedlist] (https://pypi.org/project/namedlist). The namedlist.namedlist function creates a class with __slots__ :

>>> Point = namedlist('Point', ('x', 'y', 'z'))

Another package [attrs] (https://pypi.org/project/attrs) allows you to automate the process of creating classes both with and without __slots__ .

Tuple

Python also has a built-in type tuple for representing immutable data structures. A tuple is a fixed structure or record, but without field names. For field access, the field index is used. The tuple fields are once and for all associated with the value objects at the time of creating the tuple instance:

>>> ob = (1,2,3) >>> x = ob[0] >>> ob[1] = y # ERROR

Instances of tuple are quite compact:

They occupy 8 bytes in memory more than instances of classes with __slots__ , since the tuple trace in memory also contains a number of fields:

Namedtuple

Since the tuple is used very widely, one day there was a request that you could still have access to the fields and by name too. The answer to this request was the module collections.namedtuple .

The namedtuple function is designed to automate the process of generating such classes:

>>> Point = namedtuple('Point', ('x', 'y', 'z'))

It creates a subclass of tuple, in which descriptors are defined for accessing fields by name. For our example, it would look something like this:

 class Point(tuple): # @property def _get_x(self): return self[0] @property def _get_y(self): return self[1] @property def _get_z(self): return self[2] # def __new__(cls, x, y, z): return tuple.__new__(cls, (x, y, z))

All instances of such classes have a memory footprint identical to that of a tuple. A large number of instances leave a slightly larger memory footprint:

Recordclass: mutable namedtuple without cyclic GC

Since the tuple and, accordingly, namedtuple -classes generate immutable objects in the sense that attribute ob.x can no longer be associated with another value object, a request for a mutable namedtuple variant has arisen. Since there is no built-in type in Python that is identical to the tuple that supports assignments, many options have been created. We will focus on [recordclass] (https://pypi.org/project/recordclass), which received a rating of [stackoverflow] (https://stackoverflow.com/questions/29290359/existence-of-mutable-named-tuple-in -python / 29419745). In addition it can be used to reduce the size of objects in RAM compared to the size of tuple -like objects..

The package recordclass introduces the type recordclass.mutabletuple , which is almost identical to the tuple, but also supports assignments. On its basis, subclasses are created that are almost completely identical to namedtuples, but also support the assignment of new values to fields (without creating new instances). The recordclass function, like the namedtuple function, allows you to automate the creation of these classes:

 >>> Point = recordclass('Point', ('x', 'y', 'z')) >>> ob = Point(1, 2, 3)

Class instances have same structure as tuple , but only without PyGC_Head :

By default, the recordclass function create a class that does not participate in the cyclic garbage collection mechanism. Typically, namedtuple and recordclass are used to generate classes representing records or simple (non-recursive) data structures. Using them correctly in Python does not generate circular references. For this reason, in the wake of instances of classes generated by recordclass , by default, the PyGC_Head fragment is excluded, which is necessary for classes supporting the cyclic garbage collection mechanism (more precisely: in the PyTypeObject structure, corresponding to the created class, in the flags field, by default, the flag Py_TPFLAGS_HAVE_GC` is not set).

The size of the memory footprint of a large number of instances is smaller than that of instances of the class with __slots__ :

Dataobject

Another solution proposed in the recordclass library is based on the idea: use the same storage structure in memory as in class instances with __slots__ , but do not participate in the cyclic garbage collection mechanism. Such classes are generated using the recordclass.make_dataclass function:

 >>> Point = make_dataclass('Point', ('x', 'y', 'z'))

The class created in this way, by default, creates mutable instances.

Another way – use class declaration with inheritance from recordclass.dataobject :

class Point(dataobject): x:int y:int z:int

Classes created in this way will create instances that do not participate in the cyclic garbage collection mechanism. The structure of the instance in memory is the same as in the case with __slots__ , but without the PyGC_Head :

>>> ob = Point(1,2,3) >>> print(sys.getsizeof(ob)) 40

To access the fields, special descriptors are also used to access the field by its offset from the beginning of the object, which are located in the class dictionary:

The sizeo of the memory footprint of a large number of instances is the minimum possible for CPython:

Cython

There is one approach based on the use of [Cython] (https://cython.org). Its advantage is that the fields can take on the values of the C language atomic types. Descriptors for accessing fields from pure Python are created automatically. For example:

cdef class Python: cdef public int x, y, z def __init__(self, x, y, z): self.x = x self.y = y self.z = z

In this case, the instances have an even smaller memory size:

>>> ob = Point(1,2,3) >>> print(sys.getsizeof(ob)) 32

The instance trace in memory has the following structure:

The size of the footprint of a large number of copies is less:

However, it should be remembered that when accessing from Python code, a conversion from int to a Python object and vice versa will be performed every time.

Numpy

Using multidimensional arrays or arrays of records for a large amount of data gives a gain in memory. However, for efficient processing in pure Python, you should use processing methods that focus on the use of functions from the numpy package.

>>> Point = numpy.dtype(('x', numpy.int32), ('y', numpy.int32), ('z', numpy.int32)])

An array of N elements, initialized with zeros, is created using the function:

 >>> points = numpy.zeros(N, dtype=Point)

The size of the array in memory is the minimum possible:

Normal access to array elements and rows will require convertion from a Python object to a C int value and vice versa. Extracting a single row results in the creation of an array containing a single element. Its trace will not be so compact anymore:

Therefore, as noted above, in Python code, it is necessary to process arrays using functions from the numpy package.

Conclusion

On a clear and simple example, it was possible to verify that the Python programming language (CPython) community of developers and users has real possibilities for a significant reduction in the amount of memory used by objects.

Источник

Optimizing Memory Usage in Python Applications

Find out why your Python apps are using too much memory and reduce their RAM usage with these simple tricks and efficient data structures

When it comes to performance optimization, people usually focus only on speed and CPU usage. Rarely is anyone concerned with memory consumption, well, until they run out of RAM. There are many reasons to try to limit memory usage, not just avoiding having your application crash because of out-of-memory errors.

In this article we will explore techniques for finding which parts of your Python applications are consuming too much memory, analyze the reasons for it and finally reduce the memory consumption and footprint using simple tricks and memory efficient data structures.

Why Bother, Anyway?

But first, why should you bother saving RAM anyway? Are there really any reason to save memory other than avoiding the aforementioned out-of-memory errors/crashes?

One simple reason is money. Resources — both CPU and RAM — cost money, why waste memory by running inefficient applications, if there are ways to reduce the memory footprint?

Another reason is the notion that “data has mass”, if there’s a lot of it, then it will move around slowly. If data has to be stored on disk rather than in RAM or fast caches, then it will take a while to load and get processed, impacting overall performance. Therefore, optimizing for memory usage might have a nice side effect of speeding-up the application runtime.

Lastly, in some cases performance can be improved by adding more memory (if application performance is memory-bound), but you can’t do that if you don’t have any memory left on the machine.

Find Bottlenecks

It’s clear that there are good reasons to reduce memory usage of our Python applications, before we do that though, we first need to find the bottlenecks or parts of code that are hogging all the memory.

First tool we will introduce is memory_profiler . This tool measures memory usage of specific function on line-by-line basis:

Источник