Multiprocessing Pool Exception Handling in Python
Exceptions may be raised when initializing worker processes, in target task processes, and in callback functions once tasks are completed.
In this tutorial you will discover how to handle exceptions in a Python multiprocessing pool.
Multiprocessing Pool Exception Handling
Exception handling is an important consideration when using processes.
Code may raise an exception when something unexpected happens and the exception should be dealt with by your application explicitly, even if it means logging it and moving on.
There are three points you may need to consider exception handling when using the multiprocessing.pool.Pool, they are:
Let’s take a closer look at each point in turn.
Run your loops using all CPUs, download my FREE book to learn how.
Exception Handling in Worker Initialization
You can specify a custom initialization function when configuring your multiprocessing.pool.Pool.
This can be set via the “initializer” argument to specify the function name and “initargs” to specify a tuple of arguments to the function.
Each process started by the process pool will call your initialization function before starting the process.
You can learn more about configuring the pool with worker initializer functions in the tutorial:
If your initialization function raises an exception it will break your process pool.
We can demonstrate this with an example of a contrived initializer function that raises an exception.
Running the example fails with an exception, as we expected.
The process pool is created and nearly immediately, the internal child worker processes are created and initialized.
Each worker process fails to be initialized given that the initialization function raises an exception.
The process pool then attempts to restart new replacement child workers for each process that was started and failed. These too fail with exceptions.
The process repeats many times until some internal limit is reached and the program exits.
A truncated example of the output is listed below.
This highlights that if you use a custom initializer function, that you must carefully consider the exceptions that may be raised and perhaps handle them, otherwise out at risk all tasks that depend on the process pool.
Confused by the Pool class API?
Download my FREE PDF cheat sheet
Exception Handling in Task Execution
An exception may occur while executing your task.
This will cause the task to stop executing, but will not break the process pool.
If tasks were issued with a synchronous function, such as apply(), map(), or starmap() the exception will be re-raised in the caller.
If tasks are issued with an asynchronous function such as apply_async(), map_async(), or starmap_async(), an AsyncResult object will be returned. If a task issued asynchronously raises an exception, it will be caught by the process pool and re-raised if you call get() function in the AsyncResult object in order to get the result.
It means that you have two options for handling exceptions in tasks, they are:
- Handle exceptions within the task function.
- Handle exceptions when getting results from tasks.
Let’s take a closer look at each approach in turn.
Exception Handling Within the Task
Handling the exception within the task means that you need some mechanism to let the recipient of the result know that something unexpected happened.
This could be via the return value from the function, e.g. None.
Alternatively, you can re-raise an exception and have the recipient handle it directly. A third option might be to use some broader state or global state, perhaps passed by reference into the call to the function.
The example below defines a work task that will raise an exception, but will catch the exception and return a result indicating a failure case.
Python: Multiprocessing and Exceptions
Python’s multiprocessing module provides an interface for spawning and managing child processes that is familiar to users of the threading module. One problem with the multiprocessing module, however, is that exceptions in spawned child processes don’t print stack traces:
Consider the following snippet:
import multiprocessing import somelib def f(x): return 1 / somelib.somefunc(x) if __name__ == '__main__': with multiprocessing.Pool(5) as pool: print(pool.map(f, range(5))) and the following error message:
Traceback (most recent call last): File "test.py", line 9, in print(pool.map(f, range(5))) File "/usr/lib/python3.3/multiprocessing/pool.py", line 228, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/usr/lib/python3.3/multiprocessing/pool.py", line 564, in get raise self._value ZeroDivisionError: division by zero What triggered the ZeroDivisionError ? Did somelib.somefunc(x) return 0, or did some other computation in somelib.somefunc() cause the exception? You will notice that we only see the stack trace of the main process, whereas the stack trace of the code that actually triggered the exception in the worker processes is not shown at all.
Luckily, Python provides a handy traceback module for working with exceptions and stack traces. All we have to do is catch the exception inside the worker process, and print it. Let’s change the code above to read:
import multiprocessing import traceback import somelib def f(x): try: return 1 / somelib.somefunc(x) except Exception as e: print('Caught exception in worker thread (x = %d):' % x) # This prints the type, value, and stack trace of the # current exception being handled. traceback.print_exc() print() raise e if __name__ == '__main__': with multiprocessing.Pool(5) as pool: print(pool.map(f, range(5))) Now, if you run the same code again, you will see something like this:
Caught exception in worker thread (x = 0): Traceback (most recent call last): File "test.py", line 7, in f return 1 / somelib.somefunc(x) File "/path/to/somelib.py", line 2, in somefunc return 1 / x ZeroDivisionError: division by zero Traceback (most recent call last): File "test.py", line 16, in print(pool.map(f, range(5))) File "/usr/lib/python3.3/multiprocessing/pool.py", line 228, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/usr/lib/python3.3/multiprocessing/pool.py", line 564, in get raise self._value ZeroDivisionError: division by zero The printed traceback reveals somelib.somefunc() to be the actual culprit.
In practice, you may want to save the exception and the stack trace somewhere. For that, you can use the file argument of print_exc in combination with StringIO . For example:
import logging import io # Import StringIO in Python 2 . def Work(. ): try: . except Exception as e: exc_buffer = io.StringIO() traceback.print_exc(file=exc_buffer) logging.error( 'Uncaught exception in worker process:\n%s', exc_buffer.getvalue()) raise e