Python multiprocessing get value

Multiprocessing Pool Get Result from Asynchronous Tasks

You can get results from tasks in the multiprocessing pool using a callback or by calling AsyncResult.get().

In this tutorial you will discover how to get results from tasks issued asynchronously to the multiprocessing pool in Python.

Multiprocessing Pool Asynchronous Tasks

We can execute tasks asynchronously using the multiprocessing pool.

The multiprocessing.Pool class provides three functions for issuing tasks asynchronously, they are:

• Pool.apply_async(): For executing one-off tasks.
• Pool.map_async(): For calling the same function multiple times with different arguments.
• Pool.starmap_async(): For calling the same function many times with more than one argument.

Each of these approaches for executing an asynchronous task in the multiprocessing pool returns immediately with an AsyncResult object.

This object provides a handle on the task or tasks and allows us to check on the status of the tasks and to get results.

How can we get results from asynchronous tasks?

Run your loops using all CPUs, download my FREE book to learn how.

How to Get Result from an Asynchronous Tasks

There are two ways that we can get results from asynchronous tasks executed with the multiprocessing pool.

Let’s take a closer look at each approach.

How to Get Result Via a Callback

We can get results from asynchronous tasks via a result callback.

A result callback can be specified via the “callback” argument.

The argument specifies the name of a custom function to call with the result of asynchronous task or tasks.

The function may have any name you like, as long as it does not conflict with a function name already in use.

For example, if apply_async() is configured with a callback, then the callback function will be called with the return value of the task function that was executed.

Alternatively, if map_async() or starmap_async() are configured with a callback, then the callback function will be called with an iterable of return values from all tasks issued to the process pool.

You can learn more about result callbacks in the tutorial:

How to Get Result Via AsyncResult

We can get results from asynchronous tasks via the AsyncResult object.

A AsyncResult provides a handle on one or more issued tasks.

It allows the caller to check on the status of the issued tasks, to wait for the tasks to complete, and to get the results once tasks are completed.

We can get the result of an issued task by calling the AsyncResult.get() function.

This will return the result of the specific function called to issue the task.

• apply_async(): Returns the return value of the target function.
• map_async(): Returns an iterable over the return values of the target function.
• starmap_async(): Returns an iterable over the return values of the target function.

If the issued tasks have not yet completed, then get() will block until the tasks are finished.

A “timeout” argument can be specified. If the tasks are still running and do not complete within the specified number of seconds, a multiprocessing.TimeoutError is raised.

If an issued task raises an exception, the exception will be re-raised once the issued tasks are completed.

We may need to handle this case explicitly if we expect a task to raise an exception on failure.

You can learn more about the AsyncResult object in the tutorial:

Now that we know how to get results from asynchronous tasks issued to the multiprocessing pool, let’s look at some worked examples.

Confused by the Pool class API?
Download my FREE PDF cheat sheet

How to Get Result From apply_async()

We can explore how to get results from tasks issued asynchronously with apply_async().

In this example we define a simple task that generates and returns a number. We will then issue a task to execute this function then get the result from the issued task.

Firstly, we can define a target task function.

The function will generate a random number between 0 and 1. It will then block for a fraction of a second to simulate computational effort, then return the number that was generated.

The task() function below implements this.

Next, in the main process, we will create a multiprocessing pool with the default configuration using the context manager interface.

We will then issue a call to the task() function to be executed by the multiprocessing pool asynchronously via the apply_async() function.

You can learn more about how to use the apply_async() function in the tutorial:

This call returns immediately with an AsyncResult object.

We then call the get() function to get the result from the task.

This will block until the task has completed and then returns the return value from the task.

Finally, we report the value retrieved from the task.

Tying this together, the complete example is listed below.

Running the example first creates the multiprocessing pool with a default configuration.

Next, the task is issued to the pool to be executed asynchronously.

An AsyncResult object is returned and the main process blocks attempting to get the result.

The task executes, first generating a random number, blocking for a fraction of a second, then returning the number.

The main process receives the result from the asynchronous task and reports the value.

Note, the output will differ each time the program is run given the use of random numbers.

This example demonstrated how to get the result from a task executed asynchronously with apply_async().

Next, let’s look at how we might get the result from tasks executed with map_async().

Free Python Multiprocessing Pool Course

Download my Pool API cheat sheet and as a bonus you will get FREE access to my 7-day email course.

Discover how to use the Multiprocessing Pool including how to configure the number of workers and how to execute tasks asynchronously.

How to Get Result From map_async()

We can explore how to get results from tasks issued asynchronously with map_async().

In this example we define a simple task that takes an integer as an argument, generates a random number, then returns a combination of the argument with the generated number. We will then have the multiprocessing pool execute the function multiple times asynchronously with different arguments.

Firstly, we can define a target task function.

The function will take an integer as an argument. It will generate a random number between 0 and 1. It will then block for a fraction of a second to simulate computational effort, then return the generated number added to the input argument.

The task() function below implements this.

Next, in the main process, we will create a multiprocessing pool with the default configuration using the context manager interface.

We will then issue a call to the task() function multiple times with values between 0 and 4. The tasks are issued using map_async() to be executed asynchronously.

You can learn more about how to use the map_async() function in the tutorial:

This call returns immediately with an AsyncResult object.

We then call the get() function to get the result from the issued tasks, specifically an iterator over return values from the five function calls.

This call will block until all tasks have completed.

Finally, we report the values retrieved from the five issued tasks by iterating over the return values and reporting each in turn.

Tying this together, the complete example is listed below.

Running the example first creates the multiprocessing pool with a default configuration.

Next, five tasks are issued to the pool to be executed asynchronously.

An AsyncResult object is returned and the main process blocks attempting to get the iterable of return values.

The tasks execute in parallel, each task taking an integer argument, generating a random number, blocking for a fraction of a second, then returning the generated number added to the argument.

The main process receives the result from the asynchronous task. It then traverses the returned iterable and reports the return value from each task.

Note, the output will differ each time the program is run given the use of random numbers.

This example demonstrates how to get results from tasks executed asynchronously with map_async().

Next, let’s look at how we might get the result from tasks executed with starmap_async().

Overwheled by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps

How to Get Result From starmap_async()

We can explore how to get results from tasks issued asynchronously with starmap_async().

In this example we define a simple task that takes three integers as an argument, generates a random number, then returns a combination of the input arguments with the generated number. We will then have the multiprocessing pool execute the function multiple times asynchronously with different arguments.

Firstly, we can define a target task function.

The function will take three integers as arguments. It will generate a random number between 0 and 1. It then blocks for a fraction of a second to simulate computational effort, then adds the generated number and all arguments and returns the sum.

The task() function below implements this.

Источник

Multiprocessing Return Value From Process

You can return a variable from a child process using a multiprocessing.Value or a multiprocessing.Queue.

In this tutorial you will discover how to return a value from a process in Python.

Need to Return Value From Process

A process is a running instance of a computer program.

Every Python program is executed in a Process, which is a new instance of the Python interpreter. This process has the name MainProcess and has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.

Sometimes we may need to create new child processes in our program in order to execute code concurrently.

Python provides the ability to create and manage new processes via the multiprocessing.Process class.

In multiprocessing programming we typically need to return a value from a process.

This is challenging as there are no direct methods for returning a value from a process to another calling process.

How can we return a value from a process?

Run your loops using all CPUs, download my FREE book to learn how.

How to Return Value From a Process

There are no direct methods to return a value from a process.

Instead, it can be achieved using indirect methods.

There are a number of indirect methods to choose from. The three most convenient and widely used methods to return a value from a process are as follows:

Other approaches might include:

• Use a multiprocessing.Manager, e.g. processes interact with the same object via proxies.
• Use multiprocessing.sharedctypes, e.g. same methods that underlie multiprocessing.Value.
• Use multiprocessing.shared_memory.

Do you know of any other methods?
Let me know in the comments below.

Next, let’s take a closer look at some of these methods.

Return Variable From Process with Value

We can return a variable from a process using a multiprocessing.Value object.

These classes explicitly define data attributes designed to be shared between processes in a process-safe manner.

A process-safe manner means that only one process can read or access the variable at a time. Shared variables mean that changes made in one process are always propagated and made available to other processes.

An instance of the multiprocessing.Value can be defined in the constructor of a custom class as a shared instance variable.

The constructor of the multiprocessing.Value class requires that we specify the data type and an initial value.

The data type can be specified using ctype “type” or a typecode.

You can learn more about ctypes here:

Typecodes are familiar and easy to use, for example ‘i’ for a signed integer or ‘f’ for a single floating-point value.

You can see a handy table of type codes here:

For example, we can define a Value shared memory variable that holds a signed integer and is initialized to the value zero.

Источник