printing stdout in realtime from a subprocess that requires stdin
This is a follow up to this question, but if I want to pass an argument to stdin to subprocess , how can I get the output in real time? This is what I currently have; I also tried replacing Popen with call from the subprocess module and this just leads to the script hanging.
from subprocess import Popen, PIPE, STDOUT cmd = 'rsync --rsh=ssh -rv --files-from=- thisdir/ servername:folder/' p = Popen(cmd.split(), stdout=PIPE, stdin=PIPE, stderr=STDOUT) subfolders = '\n'.join(['subfolder1','subfolder2']) output = p.communicate(input=subfolders)[0] print output In the former question where I did not have to pass stdin I was suggested to use p.stdout.readline , there there is no room there to pipe anything to stdin . Addendum: This works for the transfer, but I see the output only at the end and I would like to see the details of the transfer while it’s happening.
3 Answers 3
In order to grab stdout from the subprocess in real time you need to decide exactly what behavior you want; specifically, you need to decide whether you want to deal with the output line-by-line or character-by-character, and whether you want to block while waiting for output or be able to do something else while waiting.
It looks like it will probably suffice for your case to read the output in line-buffered fashion, blocking until each complete line comes in, which means the convenience functions provided by subprocess are good enough:
p = subprocess.Popen(some_cmd, stdout=subprocess.PIPE) # Grab stdout line by line as it becomes available. This will loop until # p terminates. while p.poll() is None: l = p.stdout.readline() # This blocks until it receives a newline. print l # When the subprocess terminates there might be unconsumed output # that still needs to be processed. print p.stdout.read() If you need to write to the stdin of the process, just use another pipe:
p = subprocess.Popen(some_cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE) # Send input to p. p.stdin.write("some input\n") p.stdin.flush() # Now start grabbing output. while p.poll() is None: l = p.stdout.readline() print l print p.stdout.read() Pace the other answer, there’s no need to indirect through a file in order to pass input to the subprocess.
subprocess.Popen.stdout — reading stdout in real-time (again)
My case is that I have a console app written in C, lets take for example this code in a loop:
tmp = 0.0; printf("\ninput>>"); scanf_s("%f",&tmp); printf ("\ninput was: %f",tmp); It continuously reads some input and writes some output.
My python code to interact with it is the following:
p=subprocess.Popen([path],stdout=subprocess.PIPE,stdin=subprocess.PIPE) p.stdin.write('12345\n') for line in p.stdout: print(">>> " + str(line.rstrip())) p.stdout.flush() So far whenever I read form p.stdout it always waits until the process is terminated and then outputs an empty string. I’ve tried lots of stuff — but still the same result.
I tried Python 2.6 and 3.1, but the version doesn’t matter — I just need to make it work somewhere.
I’m in exactly the same situation, having read all those question and still found nothing elegant that works.
5 Answers 5
Trying to write to and read from pipes to a sub-process is tricky because of the default buffering going on in both directions. It’s extremely easy to get a deadlock where one or the other process (parent or child) is reading from an empty buffer, writing into a full buffer or doing a blocking read on a buffer that’s awaiting data before the system libraries flush it.
For more modest amounts of data the Popen.communicate() method might be sufficient. However, for data that exceeds its buffering you’d probably get stalled processes (similar to what you’re already seeing?)
You might want to look for details on using the fcntl module and making one or the other (or both) of your file descriptors non-blocking. In that case, of course, you’ll have to wrap all reads and/or writes to those file descriptors in the appropriate exception handling to handle the «EWOULDBLOCK» events. (I don’t remember the exact Python exception that’s raised for these).
A completely different approach would be for your parent to use the select module and os.fork() . and for the child process to execve() the target program after directly handling any file dup()ing. (Basically you’d be re-implement parts of Popen() but with different parent file descriptor (PIPE) handling.
Incidentally, .communicate, at least in Python’s 2.5 and 2.6 standard libraries, will only handle about 64K of remote data (on Linux and FreeBSD). This number may vary based on various factors (possibly including the build options used to compile your Python interpreter, or the version of libc being linked to it). It is NOT simply limited by available memory (despite J.F. Sebastian’s assertion to the contrary) but is limited to a much smaller value.
Reading stdout from a subprocess in real time
If I run execute(cmd1) the output will be printed without any problems. However, If I run execute(cmd2) instead nothing will be printed, why is that and how can I fix it so I could see the http.server’s output in real time. Also, how for line in p.stdout is been evaluated internally? is it some sort of endless loop till reaches stdout eof or something? This topic has already been addressed few times here in SO but I haven’t found a windows solution. The above snippet is code from this answer and I’m running http.server from a virtualenv (python3.6.2-32bits on win7)
5 Answers 5
If you want to read continuously from a running subprocess, you have to make that process’ output unbuffered. Your subprocess being a Python program, this can be done by passing -u to the interpreter:
This is how it looks on a Windows box.
Works for me, although I felt no desire to fire up a Windows box for that and thus removed your cmd /c . stuff.
Streams and buffering work more or less the same on Windows and other OS. See the screenshot in my updated answer. Can you reproduce this?
With this code, you can`t see the real-time output because of buffering:
for line in p.stdout: print(line, end='') But if you use p.stdout.readline() it should work:
while True: line = p.stdout.readline() if not line: break print(line, end='') See corresponding python bug discussion for details
UPD: here you can find almost the same problem with various solutions on stackoverflow.
I think the main problem is that http.server somehow is logging the output to stderr , here I have an example with asyncio , reading the data either from stdout or stderr .
My first attempt was to use asyncio, a nice API, which exists in since Python 3.4. Later I found a simpler solution, so you can choose, both of em should work.
asyncio as solution
In the background asyncio is using IOCP — a windows API to async stuff.
# inspired by https://pymotw.com/3/asyncio/subprocesses.html import asyncio import sys import time if sys.platform == 'win32': loop = asyncio.ProactorEventLoop() asyncio.set_event_loop(loop) async def run_webserver(): buffer = bytearray() # start the webserver without buffering (-u) and stderr and stdin as the arguments print('launching process') proc = await asyncio.create_subprocess_exec( sys.executable, '-u', '-mhttp.server', stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE ) print('process started <>'.format(proc.pid)) while 1: # wait either for stderr or stdout and loop over the results for line in asyncio.as_completed([proc.stderr.readline(), proc.stdout.readline()]): print('read '.format(await line)) event_loop = asyncio.get_event_loop() try: event_loop.run_until_complete(run_df()) finally: event_loop.close() redirecting the from stdout
based on your example this is a really simple solution. It just redirects the stderr to stdout and only stdout is read.
from subprocess import Popen, PIPE, CalledProcessError, run, STDOUT import os def execute(cmd): with Popen(cmd, stdout=PIPE, stderr=STDOUT, bufsize=1) as p: while 1: print('waiting for a line') print(p.stdout.readline()) cmd2 = ["python", "-u", "-m", "http.server"] execute(cmd2)