Multiprocessing Queue in Python
You can communicate between processes with queue via the multiprocessing.Queue class.
In this tutorial you will discover how to use the process queue in Python.
Need for a Queue
A process is a running instance of a computer program.
Every Python program is executed in a Process, which is a new instance of the Python interpreter. This process has the name MainProcess and has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create new child processes in our program in order to execute code concurrently.
Python provides the ability to create and manage new processes via the multiprocessing.Process class.
In multiprocessing programming, we often need to share data between processes.
One approach to sharing data is to use a queue data structure.
Python provides a number of process-safe queues, such as the multiprocessing.Queue class.
What is the Queue and how can we use it in Python?
Run your loops using all CPUs, download my FREE book to learn how.
How to Use the Queue
A queue is a data structure on which items can be added by a call to put() and from which items can be retrieved by a call to get().
The multiprocessing.Queue provides a first-in, first-out FIFO queue, which means that the items are retrieved from the queue in the order they were added. The first items added to the queue will be the first items retrieved. This is opposed to other queue types such as last-in, first-out and priority queues.
Let’s look at how we can use the multiprocessing.Queue class.
Create a Queue
The multiprocessing.Queue can be used by first creating an instance of the class. This will create an unbounded queue by default, that is, a queue with no size limit.
A create can be created with a size limit by specifying the “maxsize” argument to a value larger than zero.
Add Items to the Queue
Once a size limited queue is full, new items cannot be added and calls to put() will block until space becomes available.
Items can be added to the queue via a call to put(), for example:
By default, the call to put() will block and will not use a timeout.
For example, the above is equivalent to the following:
The call to put() will block if the queue is full. We can choose to not block when adding items by setting the “block” argument to False. If the queue is full, then a queue.Full exception will be raised which may be handled.
This can also be achieved with the put_nowait() function that does the same thing.
Alternatively we may wish to add an item to a size limited queue and block with a timeout. This can be achieved by setting the “timeout” argument to a positive value in seconds. If an item cannot be added before the timeout expires, then a queue.Full exception will be raised which may be handled.
Get Items From The Queue
Items can be retrieved from the queue by calls to get().
By default, the call to get() will block until an item is available to retrieve from the queue and will not use a timeout. Therefore, the above call is equivalent to the following:
We can retrieve items from the queue without blocking by setting the “block” argument to False. If an item is not available to retrieve, then a queue.Empty exception will be raised and may be handled.
This can also be achieved with the get_nowait() function that does the same thing.
Alternatively, we may wish to retrieve items from the queue and block with a time limit. This can be achieved by setting the “timeout” argument to a positive value in seconds. If the timeout expires before an item can be retrieved, then a queue.Empty exception will be raised and may be handled.
Query Queue Size
The number of items in the queue can be checked by the qsize() function.
We can check if the queue contains no values via the empty() function.
We may also check if the queue is full, if it is size limited when configured.
Note, the state of the queue may change immediately after any of these queries, so care must be taken to avoid a race condition.
For example, this would not be process-safe:
Now that we know how to use the multiprocessing.Queue class, let’s look at some worked examples.
Confused by the multiprocessing module API?
Download my FREE PDF cheat sheet
Example of Using a Queue
We can explore how to use the multiprocessing.Queue class with a worked example.
In this example, we will create a producer child process that will generate ten random numbers and put them on the queue. We will also create a consumer child process that will get numbers from the queue and report their values.
The multiprocessing.Queue provides a way to allow these producer and consumer processes to communicate data with each other.
First, we can define the function to be executed by the producer process.
The task will iterate ten times in a loop. Each iteration, it will generate a new random value between 0 and 1 via the random.random() function. It will then block for that fraction of a second, then put the value on the queue.
Once the task is complete it will put the value None on the queue to signal to the consumer that there is no further work.
The producer() function below implements this by taking the queue instance as an argument.