Multiple threads in while loop
I want to execute 10 threads each time and wait for it to finish and then continue with the next 10 threads until count
Consider using the largely undocumented ThreadPool class in multiprocessing.pool with ThreadPool(processes=10) . You could use while not(all(a_thread.ready() for a_thread in results)): pass to wait for all 10 threads to be idle each iteration. See my answer to another question.
Thanks for your answer, at the moment I’m working with a queue list but i guess the advantage with a pool is that after one process of the 10 is done it starts an other instead of waiting before all 10 have finsihed. Right ?
True, but if you feed it 10 things at a time and then wait for them all to finish before feeding any more into it, you’ll be able to accomplish what you want to do.
3 Answers 3
it can easily achieved using concurrents.futures library
from concurrent.futures import ThreadPoolExecutor ip = '192.168.0.' count = 0 THREAD_COUNT = 10 def work_done(future): result = future.result() # work with your result here def main(): with ThreadPoolExecutor(THREAD_COUNT) as executor: while count here executor returns future for every task it submits. keep in mind that if you use add_done_callback() finished task from thread returns to the main thread (which would block your main thread) if you really want true parallelism then you should wait for future objects separately. here's the code snippet for that.
from concurrent.futures import ThreadPoolExecutor from concurrent.futures._base import wait futures = [] with ThreadPoolExecutor(THREAD_COUNT) as executor: while count In prefer this answer because ThreadPoolExecutor seems the best way to follow. Other answers where also correct. Thanks everyone !
There are two viable options: multiprocessing with ThreadPool as @martineau suggested and using queue . Here's an example with queue that executes requests concurrently in 10 different threads. Note that it doesn't do any kind of batching, as soon as a thread completes it picks up next task without caring the status of other workers:
import queue import threading def conn(): try: while True: ip, port = que.get_nowait() print('Connecting to <>:<>'.format(ip, port)) que.task_done() except queue.Empty: pass que = queue.Queue() for i in range(256): que.put(('192.168.0.' + str(i), 80)) # Start workers threads = [threading.Thread(target=conn) for _ in range(10)] for t in threads: t.start() # Wait que to empty que.join() # Wait workers to die for t in threads: t.join() Connecting to 192.168.0.0:80 Connecting to 192.168.0.1:80 Connecting to 192.168.0.2:80 Connecting to 192.168.0.3:80 Connecting to 192.168.0.4:80 Connecting to 192.168.0.5:80 Connecting to 192.168.0.6:80 Connecting to 192.168.0.7:80 Connecting to 192.168.0.8:80 Connecting to 192.168.0.9:80 Connecting to 192.168.0.10:80 Connecting to 192.168.0.11:80 . Threading in 'while True' loop
First of all, I've only started working with Python a month ago, so I don't have deep knowledge about anything. In a project I'm trying to collect the results of multiple (simultaneous) functions in a database, infinitely until I tell it to stop. In an earlier attempt, I successfully used multiprocessing to do what I need, but since I now need to collect all the results of those functions for a database within the main, I switched to threading instead. Basically, what I'm trying to do is:
collect1 = Thread(target=collect_data1) collect2 = Thread(target=collect_data2) send1 = Thread(target=send_data1) send2 = Thread(target=send_data2) collect = (collect1, collect2) send = (send1, send2) while True: try: for thread in collect: thread.start() for thread in collect: thread.join() for thread in send: thread.start() for thread in send: thread.join() except KeyboardInterrupt: break Now, obviously I can't just restart the threads. Nor explicitly kill them. The functions within the threads can theoretically be stopped at any point, so terminate() from multiprocessing was fine. I was thinking whether something like the following could work (at least PyCharm is fine with it, so it seems to work) or if it creates a memory leak (which I assume), because the threads are never properly closed or deleted, at least as far as I can tell from researching. Again, I'm new to Python, so I don't know anything about this aspect. Code:
while True: try: collect1 = Thread(target=collect_data1) collect2 = Thread(target=collect_data2) send1 = Thread(target=send_data1) send2 = Thread(target=send_data2) collect = (collect1, collect2) send = (send1, send2) for thread in collect: thread.start() for thread in collect: thread.join() for thread in send: thread.start() for thread in send: thread.join() except KeyboardInterrupt: break I feel like this approach seems too good to be true, especially since I've never came across a similar solution during my research. Anyways, any input is appreciated. Have a good day.
Running infinite loops using threads in python
Question: I have the algorithms in place, the problem is I need to run both these in a while loops. The condition is that I cannot exit from any of them. Now threading is a good alternative for this but I have read about the GIL, so how do I go about running two infinite loops?
from multiprocessing import Process def methodA(): while TRUE: do something def methodB(): while TRUE: do something p=Process(target=methodA()) p.start() p1=Process(target=methodB()) p1.start() Now when I start process p it starts executing, after that how do I start p1 to run simultaneously?
Also, once I start executing a thread, how do I execute the other thread when the first thread is running continuously/infinitely?
I don't understand what you mean. There are two different threads executing in parallel. Unless one is dependent on the other, you have no real issue.
One method runs infinitely, I start it in a thread, now how do I run another thread?. Wait, I shall add some code to shed light on the problem.
2 Answers 2
As far as I understood your question, you have two different tasks that you want them to perform continuously. Now regarding your questions:
how do I go about running two infinite loops?
You can create two different threads that will run these infinite loops for you. The first thread will perform your task1 and second one will perform task2.
Also, once I start executing a thread, how do I execute the other thread when the first thread is running continuously/infinitely?
If you are using two different threads then you don't need to be worried about this issue. If the threads are not sharing any resource then you don't need to worry about this fact. How ever if you want to stop/pause one thread from the other thread or vice versa then you can implement a mechanism using flags or locks. These questions will help in this case:
Sample example using threading:
from threading import Thread class myClassA(Thread): def __init__(self): Thread.__init__(self) self.daemon = True self.start() def run(self): while True: print 'A' class myClassB(Thread): def __init__(self): Thread.__init__(self) self.daemon = True self.start() def run(self): while True: print 'B' myClassA() myClassB() while True: pass what if I don't want to run it using classes? How do I do this using only methods?
from threading import Thread def runA(): while True: print 'A\n' def runB(): while True: print 'B\n' if __name__ == "__main__": t1 = Thread(target = runA) t2 = Thread(target = runB) t1.setDaemon(True) t2.setDaemon(True) t1.start() t2.start() while True: pass Python - Threading and a While True Loop
I have a thread that appends rows to self.output and a loop that runs until self.done is True (or the max execution time is reached). Is there a more efficient way to do this other than using a while loop that constantly checks to see if it's done. The while loop causes the CPU to spike to 100% while it's running..
time.clock() while True: if len(self.output): yield self.output.pop(0) elif self.done or 15 < time.clock(): if 15 < time.clock(): yield "Maximum Execution Time Exceeded %s seconds" % time.clock() break 5 Answers 5
Are your threads appending to self.output here, with your main task consuming them? If so, this is a tailor-made job for Queue.Queue. Your code should become something like:
import Queue # Initialise queue as: queue = Queue.Queue() Finished = object() # Unique marker the producer will put in the queue when finished # Consumer: try: while True: next_item = self.queue.get(timeout=15) if next_item is Finished: break yield next_item except Queue.Empty: print "Timeout exceeded" Your producer threads add items to the queue with queue.put(item)
[Edit] The original code has a race issue when checking self.done (for example multiple items may be appended to the queue before the flag is set, causing the code to bail out at the first one). Updated with a suggestion from ΤΖΩΤΖΙΟΥ - the producer thread should instead append a special token (Finished) to the queue to indicate it is complete.
Note: If you have multiple producer threads, you'll need a more general approach to detecting when they're all finished. You could accomplish this with the same strategy - each thread a Finished marker and the consumer terminates when it sees num_threads markers.
Parallel while Loops in Python
I'm pretty new to Python, and programming in general and I'm creating a virtual pet style game for my little sister. Is it possible to run 2 while loops parallel to each other in python? eg:
while 1: input_event_1 = gui.buttonbox( msg = 'Hello, what would you like to do with your Potato Head?', title = 'Main Screen', choices = ('Check Stats', 'Feed', 'Exercise', 'Teach', 'Play', 'Go to Doctor', 'Sleep', 'Change Favourite Thing', 'Get New Toy', 'Quit')) if input_event_1 == 'Check Stats': myPotatoHead.check_p_h_stats() elif input_event_1 == 'Feed': myPotatoHead.feed_potato_head() elif input_event_1 == 'Exercise': myPotatoHead.exercise_potato_head() elif input_event_1 == 'Teach': myPotatoHead.teach_potato_head(myPotatoHead) elif input_event_1 == 'Play': myPotatoHead.play_with_toy() elif input_event_1 == 'Sleep': myPotatoHead.put_p_h_asleep() elif input_event_1 == 'Go to Doctor': myPotatoHead.doctor_check_up() elif input_event_1 == 'Change Favourite Thing': myPotatoHead.change_favourite_thing() elif input_event_1 == 'Quit': input_quit = gui.ynbox( msg = 'Are you sure you want to quit?', title = 'Confirm quit', choices = ('Quit', 'Cancel')) if input_quit == 1: sys.exit(0) while 1: time.sleep(20) myPotatoHead.hunger = str(float(myPotatoHead.hunger) + 1.0) myPotatoHead.happiness = str(float(myPotatoHead.happiness) - 1.0) myPotatoHead.tiredness = str(float(myPotatoHead.tiredness) + 1.0) If not, is there some way that I can turn this into one loop? I want the stuff in the second loop to happen every 20 seconds, but the stuff in the first loop to by constantly happening. Thanks for any help