Python multiprocessing pool arguments

Multiprocessing Pool map() Multiple Arguments

The multiprocessing pool map() function cannot be used directly with a target function that takes multiple arguments.

Instead, you need to use an alternate function like starmap() or a workaround like a wrapper function.

In this tutorial you will discover how to call the multiprocessing pool map() function with multiple arguments indirectly and how to use alternate approaches to execute target functions that take multiple arguments.

Need to Use Pool.map() With Multiple Arguments

The built-in map() function will call a function to each item on an iterable.

A benefit of the built-in map() function is that it can take multiple iterables. Each iterable will be traversed in lock-step, yielding an argument to the function.

The multiprocessing.Pool class provides a parallel version of the map() function. Each call to the target function will be issued to the multiprocessing pool and executed by a child worker process.

You can learn more about how to use the multiprocessing pool map() function in the tutorial:

A limitation of the multiprocessing pool map() function is that it only takes a single iterable of arguments for the target function.

This means it can only be used to call target functions that take one argument.

How can we use the multiprocessing pool map() function with multiple arguments?

Run your loops using all CPUs, download my FREE book to learn how.

How to Use Pool.map() With Multiple Arguments

There are many ways that we can use the multiprocessing pool map() function with a target function that takes multiple arguments.

We will look at 4 common approaches, they are:

1. Use Pool.apply_async() instead.
2. Use Pool.starmap() instead.
3. Change the target function to unpack arguments.
4. Use a wrapper function to unpack arguments.

Let’s take a closer look at each approach in turn.

Use Pool.apply_async() Instead

The multiprocessing pool apply() function will execute a single function call in the process pool.

Importantly, it allows a function to be called that takes zero, one, or multiple arguments. As such, it can be used in a loop instead of the map() function.

You can learn more about the multiprocessing apply() function in the tutorial:

A limitation of apply() is that it will block for each call until the target function is complete. As such, it offers no benefit of concurrency.

Instead, we can use the apply_async() function, which is an asynchronous version of the apply() function. We can specify multiple arguments to our target function using the “args” keyword and providing a tuple of arguments.

Источник

How to Configure the Multiprocessing Pool in Python

You can configure the process pool via arguments to the multiprocessing.pool.Pool class constructor.

In this tutorial you will discover how to configure the process pool in Python.

Need to Configure the Process Pool

The multiprocessing.pool.Pool in Python provides a pool of reusable processes for executing ad hoc tasks.

A process pool can be configured when it is created, which will prepare the child workers.

A process pool object which controls a pool of worker processes to which jobs can be submitted. It supports asynchronous results with timeouts and callbacks and has a parallel map implementation.

— multiprocessing — Process-based parallelism

We can issue one-off tasks to the process pool using functions such as apply() or we can apply the same function to an iterable of items using functions such as map(). Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the functions such as apply_async() and map_async().

The process pool can be customized for the application.

What can be configured in the process pool and how can we configure it?

Run your loops using all CPUs, download my FREE book to learn how.

How to Configure the Process Pool

The process pool can be configured by specifying arguments to the multiprocessing.pool.Pool class constructor.

Process Pool Arguments

The arguments to the constructor are as follows:

• processes: Maximum number of worker processes to use in the pool.
• initializer: Function executed after each worker process is created.
• initargs: Arguments to the worker process initialization function.
• maxtasksperchild: Limit the maximum number of tasks executed by each worker process.
• context: Configure the multiprocessing context such as the process start method.

Next, let’s look at the default configuration for the process pool.

Default Configuration

By default the multiprocessing.pool.Pool class constructor does not take any arguments.

Источник