Join a Multiprocessing Pool in Python
You can join a process pool by calling join() on the pool after calling close() or terminate() in order to wait for all processes in the pool to be shutdown.
In this tutorial you will discover how to join a process pool in Python.
Need to Wait for Process Pool to Close
The multiprocessing.pool.Pool in Python provides a pool of reusable processes for executing ad hoc tasks.
A process pool can be configured when it is created, which will prepare the child workers.
A process pool object which controls a pool of worker processes to which jobs can be submitted. It supports asynchronous results with timeouts and callbacks and has a parallel map implementation.
— multiprocessing — Process-based parallelism
We can issue one-off tasks to the process pool using functions such as apply() or we can apply the same function to an iterable of items using functions such as map(). Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the functions such as apply_async() and map_async().
The process pool must be shutdown once we are finished with it in order to release the child worker processes.
We often need to wait for the process pool to close completely and release all resources before continuing on in our application.
How can we safely know when the process pool is shut down completely?
Run your loops using all CPUs, download my FREE book to learn how.
What is Joining?
Joining a resource is a common pattern in concurrent programming.
It is a mechanism on an active concurrency primitive that allows the caller to wait for the target primitive to finish.
It is implemented using a join() function on the target object.
This pattern is used with thread-based and process-based concurrency.
Joining Threads
For example, it is common for one thread to call join() another target thread to wait for the target thread to finish before continuing on in the application.
How to Join a Process in Python
You can join a process by calling the Process.join() function.
In this tutorial you will discover how to join child processes in Python.
Need to Join a Process
A process is a running instance of a computer program.
Every Python program is executed in a Process, which is a new instance of the Python interpreter. Each Python process has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create additional processes in our program in order to execute code concurrently.
Python provides the ability to create and manage new processes via the multiprocessing.Process class.
You can learn more about multiprocessing in the tutorial:
In concurrent programming, we may need to wait until another process has finished running. This may be for many reasons, such as:
- The current process needs a result from the target process.
- A resource is shared between the processes.
- The current process has no other work to complete.
The join() method provides a way for one process to block until another process has finished.
How can we use the join() method to join a process in Python?
Run your loops using all CPUs, download my FREE book to learn how.
How to Join a Process
A process can be joined in Python by calling the join() method on the process instance.
This has the effect of blocking the current process until the target process that has been joined has terminated.
The target process that is being joined may terminate for a number of reasons, such as:
- Finishes executing its target function.
- Finishes executing its run() method if it extends the Process class.
- Raised an error or exception.
Once the target process has finished, the join() method will return and the current process can continue to execute.
The join() method requires that you have a multiprocessing.Process instance for the process you wish to join.
This means that if you created the process, you may need to keep a reference to the multiprocessing.Process instance.
Alternatively, you may use a multiprocessing module function to get an instance for a process, such as multiprocessing.active_children() or multiprocessing.parent_process().
The join() method also takes a “timeout” argument that specifies how long the current process is willing to wait for the target process to terminate, in seconds.
Once the timeout has expired and the target process has not terminated, the join() process will return.