Python run in threadpool

ThreadPool Example in Python

The ThreadPool is a flexible and powerful thread pool for executing ad hoc tasks in an asynchronous manner.

In this tutorial, you will discover a ThreadPool example that you can use as a template for your own project.

How to Scan Ports on a Server One-by-One (slow)

We can connect to other computers by opening a socket, called socket programming.

Opening a socket requires both the name or IP address of the server and a port number on which to connect.

For example, when your web browser opens a web page on python.org, it is opening a socket connection to that server on port 80, then using the HTTP protocol to request and download (GET) an HTML file.

Socket programming or network programming is a lot of fun.

A good first socket programming project is to develop a port scanner.

This is a program that reports all of the open sockets on a given server.

A simple way to implement a port scanner is to loop over all the ports you want to test and attempt to make a socket connection on each. If a connection can be made, we disconnect immediately and report that the port on the server is open.

For example, we know that port 80 is open on python.org, but what other ports might be open?

Historically, having many open ports on a server was a security risk, so it is common to lock down a public-facing server and close all non-essential ports to external traffic. This means scanning public servers will likely yield few open ports in the best case or will deny future access in the worst case if the server thinks you’re trying to break in.

As such, although developing a port scanner is a fun socket programming exercise, we must be careful in how we use it and what servers we scan.

Next, let’s look at how we can open a socket connection on a single port.

Open a Socket Connection on a Port

Python provides socket communication in the socket module.

A socket must first be configured in terms of the type of host address and type of socket we will create, then the configured socket can be connected.

You can learn more about the socket module in Python here:

There are many ways to specify a host address, although perhaps the most common is the IP address (IPv4) or the domain name resolved by DNS. We can configure a socket to expect this type of address via the AF_INET constant.

There are also different socket types, the most common being a TCP or stream type socket and a less reliable UDP type socket. We will attempt to open TCP sockets in this case, as they are more commonly used for services like email, web, FTP, and so on. We can configure our socket for TCP using the SOCK_STREAM constant.

We can create and configure our socket as follows:

Источник

How to Configure the ThreadPool in Python

You can configure the thread pool via arguments to the multiprocessing.pool.ThreadPool class constructor.

In this tutorial you will discover how to configure the ThreadPool in Python.

Need to Configure the ThreadPool

The multiprocessing.pool.ThreadPool in Python provides a pool of reusable threads for executing ad hoc tasks.

A thread pool object which controls a pool of worker threads to which jobs can be submitted.

— multiprocessing — Process-based parallelism

The ThreadPool class extends the Pool class. The Pool class provides a pool of worker processes for process-based concurrency.

Although the ThreadPool class is in the multiprocessing module it offers thread-based concurrency and is best suited to IO-bound tasks, such as reading or writing from sockets or files.

A ThreadPool can be configured when it is created, which will prepare the new threads.

We can issue one-off tasks to the ThreadPool using functions such as apply() or we can apply the same function to an iterable of items using functions such as map().

Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the functions such as apply_async() and map_async().

The ThreadPool can be customized for the application.

What can be configured in the ThreadPoolpool and how can we configure it?

Run your loops using all CPUs, download my FREE book to learn how.

How to Configure the ThreadPool

The ThreadPool can be configured by specifying arguments to the multiprocessing.pool.ThreadPool class constructor.

ThreadPool Constructor Arguments

The arguments to the constructor are as follows:

• processes: Maximum number of worker threads (not processes) to use in the pool.
• initializer: Function executed after each worker thread is created.
• initargs: Arguments to the worker thread initialization function.

Unlike the multiprocessing.pool.Pool class that the ThreadPool extends, the ThreadPool does not have a “maxtasksperchild” argument to limit the number of tasks per worker. Also, because we are using threads instead of processes, we cannot configure the multiprocessing “context” used by the pool.

Next, let’s look at the default configuration for the ThreadPool.

Default Configuration

By default the multiprocessing.pool.ThreadPool class constructor does not take any arguments.

Источник

Python Threadpool

With the passage of time, the data involved in a program has increased exponentially, and this has led to the adaptation of new techniques, which reduces the execution time of a program. Out of this need for faster program execution, the concept of Concurrency came into being. In this topic, we are going to learn about Python Threadpool.

Web development, programming languages, Software testing & others

The literal meaning of the word Concurrency is a simultaneous occurrence. Likewise, the concept of Concurrency is about parallel computation, and thus it decreases the execution time of a program. In Python, there are mainly three simultaneously occurring entities, namely thread, task, and processes.

Before discussing the main topic, let us first have a brief understanding of Threads and Threading.

Threads: A Thread is a component of a Process that can run parallely. There can be multiple threads inside a parent process. All the threads share the program to be executed and the data required for it within the parent process.

Threading: Threading is a library in Python that helps to achieve parallel programming with the various threads residing inside the parent process.

Working of Threading

Threading’s fundamental unit is a thread, multiple of which can reside inside a parent process, and each one accomplishes a separate task. The thread object first needs to be created and initialized by passing the function name and the arguments.

Then to start a particular thread, the start() function is required, and the join() function indicates that the execution of that thread is complete. Each thread needs its separate session for execution. Since request.Session() is not thread-safe; one thread can access a piece of code or memory at one time, and it is achieved by threading.Lock.

Thread Pool in Python

In Python, a Thread Pool is a group of idle threads pre-instantiated and are ever ready to be given the task.

We can either instantiate new threads for each or use Python Thread Pool for new threads. But when the number of tasks is way more than Python Thread Pool is preferred over the former method.

A thread pool can manage parallel execution of a large number of threads as follows: –

• A thread can be reused if a thread in a thread pool completes its execution.
• A new thread is created to replace a thread that is terminated.

How to use Python Threadpool?

concurrent.futures is a module present in the Python standard library. It contains a concrete subclass known as ThreadPoolExecuter, which uses multi-threading, and we get a pool of thread for submitting the tasks. The pool thus created assigns tasks to the available threads and scheduled them to run.

Let us see the syntax of Thread Pool Executor to better understand its working: –

from concurrent.futures import ThreadPoolExecutor from time import sleep def func_name(arguements): // function definition def main(): executor = ThreadPoolExecutor(num_of_threads) future = executor.submit(function_name, (arguement)) print(future.done()) sleep(2) print(future.done()) print(future.result()) if __name__ == '__main__': main()

Examples of Python Threadpool

Here are the Examples of Python Threadpool mention below

Example #1

from concurrent.futures import ThreadPoolExecutor from time import sleep def count_number_of_words(sentence): number_of_words = len(sentence.split()) sleep(1) print("Number of words in the sentence :",sentence," : <>".format(number_of_words),end="\n") def count_number_of_characters(sentence): number_of_characters = len(sentence) sleep(1) print("Number of characters in the sentence :",sentence," : <>".format(number_of_characters),end="\n") if __name__ == '__main__': sentence = "Python Multiprocessing is an important library for achieving parallel programming." executor = ThreadPoolExecutor(4) thread1 = executor.submit(count_number_of_words, (sentence)) thread2 = executor.submit(count_number_of_characters, (sentence)) print("Thread 1 executed ? :",thread1.done()) print("Thread 2 executed ? :",thread2.done()) sleep(2) print("Thread 1 executed ? :",thread1.done()) print("Thread 2 executed ? :",thread2.done())

Code Explanation: In the above example, a Thread Pool Executor has been created with 4 threads. Then two tasks which are signified by the functions count_number_of_words and count_number_of_characters, respectively, will wait for 1 second each before executing the functions and displaying the result. The tasks do not complete in the first one-second interval, so the call to the done() function returns a False value. After the task is executed and the respective print statements are displayed, then again, when the done() function is called, it returns a true value.

Example #2

import concurrent.futures import numpy as np from time import sleep numbers = [10,23,54,7,89,100] def get_max_number(numbers): greatest_num = np.max(numbers) sleep(2) print("Greatest number is :<>".format(greatest_num)) with concurrent.futures.ThreadPoolExecutor(max_workers = 4) as executor: thread1 = executor.submit(get_max_number, (numbers)) print("Thread 1 executed ? :",thread1.done()) sleep(2) print("Thread 1 executed ? :",thread1.done())

Code Explanation: This example shows the Context Manager’s use to instantiate the ThreadPoolExecuter, with the help of which we have created 4 threads. Then the task, which is signified by the function get_max_number(arguments), will wait for 2 seconds before executing the function and displaying the result. The tasks do not complete in the first two-second interval, so the call to the done() function returns a False value. After the task is executed and the respective print statements are displayed, then again, when the done() function is called, it returns a true value.

Example #3

from concurrent.futures import ThreadPoolExecutor from concurrent.futures import as_completed import numpy as np from time import sleep def log(n): log_value = np.log(n) sleep(1) return log_value if __name__ == '__main__': values = [1,10,100,1000] with ThreadPoolExecutor(max_workers = 3) as executor: thread1 = executor.map(log, values) for result in thread1: print(np.round(result,2))

Code Explanation: This example shows the use of the Executor.map function has been displayed. We know in Python, a map function is used to apply a certain function to every element within iterables. Here we have mapped all the elements of the iterator values to the function named log(argument) and have submitted these as independent jobs to the ThreadPoolExecutor.

Conclusion

In the following article, we have discussed the fundamentals of Python Threadpool and how it works internally. We saw at the syntax of Python Thread Pool along with 3 examples to better understand the concept. So next time you stumble upon a program that requires parallel computation, does remember to use threads and use ThreadPoolExecutor to better appreciate the use of them.

Recommended Articles

This is a guide to Python Threadpool. Here we discuss the basic concept, how to use a Python Threadpool? along with the respective examples. You may also have a look at the following articles to learn more –

Источник