Non-blocking read from stdin in python
Non-blocking I/O on python seems not so well documented. It took me some time (and several searches) to figure this out.
Below are two solutions: the first using select(), which is only available on unix, and the second using only threading and queues, which also works on windows.
The Unix way: select()
The simpler solution is using select on sys.stdin, reading input when available and doing whatever when not available. Unfortunately, due to limitations of the select module, this solution does not work on windows because you cannot poll stdin with select.select().
#! /usr/bin/python3 """Treat input if available, otherwise exit without blocking""" import sys import select def something(line): print('read input:', line, end='') def something_else(): print('no input') # If there's input ready, do something, else do something # else. Note timeout is zero so select won't block at all. while sys.stdin in select.select([sys.stdin], [], [], 0)[0]: line = sys.stdin.readline() if line: something(line) else: # an empty line means stdin has been closed print('eof') exit(0) else: something_else()
$ echo -e 'a\nb' | nonblocking.py read input: a read input: b eof $ time nonblocking.py no input nonblocking.py 0.03s user 0.01s system 89% cpu 0.048 total
A slightly more elaborate example where you do something else while you wait for more input:
#! /usr/bin/python3 """Check for input every 0.1 seconds. Treat available input immediately, but do something else if idle.""" import sys import select import time # files monitored for input read_list = [sys.stdin] # select() should wait for this many seconds for input. # A smaller number means more cpu usage, but a greater one # means a more noticeable delay between input becoming # available and the program starting to work on it. timeout = 0.1 # seconds last_work_time = time.time() def treat_input(linein): global last_work_time print("Workin' it!", linein, end="") time.sleep(1) # working takes time print('Done') last_work_time = time.time() def idle_work(): global last_work_time now = time.time() # do some other stuff every 2 seconds of idleness if now - last_work_time > 2: print('Idle for too long; doing some other stuff.') last_work_time = now def main_loop(): global read_list # while still waiting for input on at least one file while read_list: ready = select.select(read_list, [], [], timeout)[0] if not ready: idle_work() else: for file in ready: line = file.readline() if not line: # EOF, remove file from input list read_list.remove(file) elif line.rstrip(): # optional: skipping empty lines treat_input(line) try: main_loop() except KeyboardInterrupt: pass
Now, say the work takes a while, and if the user interrupts, you have to do something else with the unprocessed input. Then, you can read the input in a separate thread from the work (so it will read new input even while work is being done), and save it in a queue. You work on the queue until the user interrupts, and then you stop the work and do something else with what’s left:
#! /usr/bin/python3 """Store input in a queue as soon as it arrives, and work on it as soon as possible. Do something with untreated input if the user interrupts. Do other stuff if idle waiting for input.""" import sys import select import time import threading import queue # files monitored for input read_list = [sys.stdin] # select() should wait for this many seconds for input. timeout = 0.1 # seconds last_work_time = time.time() def treat_input(linein): global last_work_time print("Workin' it!", linein, end='') time.sleep(1) # working takes time print('Done') last_work_time = time.time() def idle_work(): global last_work_time now = time.time() # do some other stuff every 2 seconds of idleness if now - last_work_time > 2: print('Idle for too long; doing some other stuff.') last_work_time = now # some sort of cleanup that involves the input def cleanup(): print() while not input_queue.empty(): line = input_queue.get() print("Didn't get to work on this line:", line, end='') # will hold input input_queue = queue.Queue() # will signal to the input thread that it should exit: # the main thread acquires it and releases on exit interrupted = threading.Lock() interrupted.acquire() # input thread's work: stuff input in the queue until # there's either no more input, or the main thread exits def read_input(): while (read_list and not interrupted.acquire(blocking=False)): ready = select.select(read_list, [], [], timeout)[0] for file in ready: line = file.readline() if not line: # EOF, remove file from input list read_list.remove(file) elif line.rstrip(): # optional: skipping empty lines input_queue.put(line) print('Input thread is done.') input_thread = threading.Thread(target=read_input) input_thread.start() try: while True: # if finished reading input and all the work is done, # exit if input_queue.empty() and not input_thread.is_alive(): break else: try: treat_input(input_queue.get(timeout=timeout)) except queue.Empty: idle_work() except KeyboardInterrupt: cleanup() # make input thread exit as well, if still running interrupted.release() print('Main thread is done.')
The portable way: letting the main thread block
As mentioned above, we cannot use select.select() on stdin on windows, which means there’s no easy way to perform non-blocking reads that works on all OSs. We can work around this, by using a queue to hold input like in the examples above, but letting the thread that is reading input block as usual.
If we expect stdin to be closed at some point and just want to perform work while waiting for input, there’s nothing else that needs to be done, we just replace our calls to select() with regular reads. The follow example reproduces the third example above using blocking reads.
Windows users should note that you might not be able to interrupt the program with Ctrl-C if stdin is not the terminal, e.g. if you piped some output into it.
#! /usr/bin/python3 """Store input in a queue as soon as it arrives, and work on it as soon as possible. Do something with untreated input if the user interrupts. Do other stuff if idle waiting for input.""" import sys import time import threading import queue timeout = 0.1 # seconds last_work_time = time.time() def treat_input(linein): global last_work_time print('Working on line:', linein, end='') time.sleep(1) # working takes time print('Done working on line:', linein, end='') last_work_time = time.time() def idle_work(): global last_work_time now = time.time() # do some other stuff every 2 seconds of idleness if now - last_work_time > 2: print('Idle for too long; doing some other stuff.') last_work_time = now def input_cleanup(): print() while not input_queue.empty(): line = input_queue.get() print("Didn't get to work on this line:", line, end='') # will hold all input read, until the work thread has chance # to deal with it input_queue = queue.Queue() # will signal to the work thread that it should exit when # it finishes working on the currently available input no_more_input = threading.Lock() no_more_input.acquire() # will signal to the work thread that it should exit even if # there's still input available interrupted = threading.Lock() interrupted.acquire() # work thread' loop: work on available input until main # thread exits def treat_input_loop(): while not interrupted.acquire(blocking=False): try: treat_input(input_queue.get(timeout=timeout)) except queue.Empty: # if no more input, exit if no_more_input.acquire(blocking=False): break else: idle_work() print('Work loop is done.') work_thread = threading.Thread(target=treat_input_loop) work_thread.start() # main loop: stuff input in the queue until there's either # no more input, or the program gets interrupted try: for line in sys.stdin: if line: # optional: skipping empty lines input_queue.put(line) # inform work loop that there will be no new input and it # can exit when done no_more_input.release() # wait for work thread to finish work_thread.join() except KeyboardInterrupt: interrupted.release() input_cleanup() print('Main loop is done.')
C:\>python nonblocking.py Idle for too long; doing some other stuff. Idle for too long; doing some other stuff. a Working on line: a Done working on line: a ^C Main loop is done. Work loop is done.
Output editted for readability.
If, on the other hand, you want to exit before stdin is closed, you must find a way to stop the blocked thread. Since _thread.interrupt_main() does nothing and, more generally, there’s no way to interrupt a thread from another one or to raise exceptions across threads (and sys.exit() only exits the thread calling it), we must fire an interrupt signal at our own process via os.kill(), which will cause the main thread to stop blocking and handle it.
Note that this has only become available on windows in version 3.2, so this method will still not work for you if you have an older version of python. Another possible method is to use os._exit(), but it might produce unwanted side-effects (see the documentation).
This following example reproduces the first example in this post, without select(), and with a timeout of 0.1 seconds.
#! /usr/bin/python3 """Treat input if available, otherwise exit without blocking""" import sys import threading import queue import os import signal timeout = 0.1 # seconds def treat_input(linein): print('read input:', linein, end='') def no_input(): print('no more input') # stop main thread (which is probably blocked reading # input) via an interrupt signal # only available for windows in version 3.2 or higher os.kill(os.getpid(), signal.SIGINT) exit() # things to be done before exiting the main thread should go # in here def cleanup(*args): exit() # handle sigint, which is being used by the work thread to # tell the main thread to exit signal.signal(signal.SIGINT, cleanup) # will hold all input read, until the work thread has chance # to deal with it input_queue = queue.Queue() # work thread's loop: work on available input until main # thread exits def treat_input_loop(): while True: try: treat_input(input_queue.get(timeout=timeout)) except queue.Empty: no_input() work_thread = threading.Thread(target=treat_input_loop) work_thread.start() # main loop: stuff input in the queue for line in sys.stdin: input_queue.put(line) # wait for work thread to finish work_thread.join()
C:\>timeit python nonblocking.py no more input Elapsed Time: 0:00:00.170 Process Time: 0:00:00.060 C:\>echo a | python nonblocking.py read input: a no more input
Output trimmed for readability.
You should take into consideration that since there’s an unavoidable race condition between the main thread storing input into the queue and the work thread reading from the queue, a long enough timeout is needed for the program not to exit immediately even if there’s input available. During testing, the smallest timeout I was able to use successfully was 10 microseconds (10^-5), but YMMV. The 100 milliseconds used in the example should be plenty, though.
Thanks to Stuart Axon for pointing out the problem with select() and to the authors of this answer on how to time things in windows and this blog post on copying the output.