Python subprocess read line by line

read subprocess stdout line by line

My python script uses subprocess to call a linux utility that is very noisy. I want to store all of the output to a log file and show some of it to the user. I thought the following would work, but the output doesn’t show up in my application until the utility has produced a significant amount of output.

#fake_utility.py, just generates lots of output over time import time i = 0 while True: print hex(i)*512 i += 1 time.sleep(0.5) #filters output import subprocess proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE) for line in proc.stdout: #the real code does filtering here print "test:", line.rstrip()

The behavior I really want is for the filter script to print each line as it is received from the subprocess. Sorta like what tee does but with python code.

What am I missing? Is this even possible?

If a sys.stdout.flush() is added to fake_utility.py, the code has the desired behavior in python 3.1. I’m using python 2.6. You would think that using proc.stdout.xreadlines() would work the same as py3k, but it doesn’t.

Here is the minimal working code.

#fake_utility.py, just generates lots of output over time import sys, time for i in range(10): print i sys.stdout.flush() time.sleep(0.5) #display out put line by line import subprocess proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE) #works in python 3.0+ #for line in proc.stdout: for line in iter(proc.stdout.readline,''): print line.rstrip()

Advertisement

Answer

I think the problem is with the statement for line in proc.stdout , which reads the entire input before iterating over it. The solution is to use readline() instead:

#filters output import subprocess proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE) while True: line = proc.stdout.readline() if not line: break #the real code does filtering here print "test:", line.rstrip()

Of course you still have to deal with the subprocess’ buffering.

Note: according to the documentation the solution with an iterator should be equivalent to using readline() , except for the read-ahead buffer, but (or exactly because of this) the proposed change did produce different results for me (Python 2.5 on Windows XP).

Источник

How to read subprocess stdout line by line in Python?

Sometimes, we want to read subprocess stdout line by line in Python?

In this article, we’ll look at how to read subprocess stdout line by line in Python?

How to read subprocess stdout line by line in Python?

To read subprocess stdout line by line in Python, we can call stdout.readline on the returned process object.

import subprocess proc = subprocess.Popen(['ls', '-l'], stdout=subprocess.PIPE) while True: line = proc.stdout.readline() if not line: break print(line.rstrip()) 

We call subprocess.Popen with a list of strings with the command and command line arguments.

Then we set stdout to subprocess.PIPE to return the output.

Next, we call proc.stdout.readline to return the next line of the stdout output in the while loop.

If line is None , then we stop the loop.

Otherwise, we print the line .

Therefore, we get text like:

b'total 64' b'-rw-r--r-- 1 runner runner 183 Oct 20 01:10 main.py' b'-rw-r--r-- 1 runner runner 14924 Oct 19 23:40 poetry.lock' b'drwxr-xr-x 1 runner runner 126 Oct 19 23:17 __pycache__' b'-rw-r--r-- 1 runner runner 319 Oct 19 23:39 pyproject.toml' b'-rw-r--r-- 1 runner runner 12543 Oct 20 00:16 somepic.png' b'-rw-r--r-- 1 runner runner 197 Oct 19 23:21 strings.json' b'-rw------- 1 runner runner 18453 Oct 20 00:16 test1.png' 

Conclusion

To read subprocess stdout line by line in Python, we can call stdout.readline on the returned process object.

Источник

read subprocess stdout line by line

I think the problem is with the statement for line in proc.stdout , which reads the entire input before iterating over it. The solution is to use readline() instead:

#filters output import subprocess proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE) while True: line = proc.stdout.readline() if not line: break #the real code does filtering here print "test:", line.rstrip() 

Of course you still have to deal with the subprocess’ buffering.

Note: according to the documentation the solution with an iterator should be equivalent to using readline() , except for the read-ahead buffer, but (or exactly because of this) the proposed change did produce different results for me (Python 2.5 on Windows XP).

Similar question

I was having a problem with the arg list of Popen to update servers, the following code resolves this a bit.

import getpass from subprocess import Popen, PIPE username = 'user1' ip = '127.0.0.1' print ('What is the password?') password = getpass.getpass() cmd1 = f"""sshpass -p ssh @""" cmd2 = f"""echo | sudo -S apt update""" cmd3 = " && " cmd4 = f"""echo | sudo -S apt upgrade -y""" cmd5 = " && " cmd6 = "exit" commands = [cmd1, cmd2, cmd3, cmd4, cmd5, cmd6] command = " ".join(commands) cmd = command.split() with Popen(cmd, stdout=PIPE, bufsize=1, universal_newlines=True) as p: for line in p.stdout: print(line, end='') 

And to run the update on a local computer, the following code example does this.

import getpass from subprocess import Popen, PIPE print ('What is the password?') password = getpass.getpass() cmd1_local = f"""apt update""" cmd2_local = f"""apt upgrade -y""" commands = [cmd1_local, cmd2_local] with Popen(['echo', password], stdout=PIPE) as auth: for cmd in commands: cmd = cmd.split() with Popen(['sudo','-S'] + cmd, stdin=auth.stdout, stdout=PIPE, bufsize=1, universal_newlines=True) as p: for line in p.stdout: print(line, end='') 

Stan S. 635

The following modification of Rômulo’s answer works for me on Python 2 and 3 (2.7.12 and 3.6.1):

import os import subprocess process = subprocess.Popen(command, stdout=subprocess.PIPE) while True: line = process.stdout.readline() if line != '': os.write(1, line) else: break 

Some more answer related to the same question

I tried this with python3 and it worked, source

def output_reader(proc): for line in iter(proc.stdout.readline, b''): print('got line: '.format(line.decode('utf-8')), end='') def main(): proc = subprocess.Popen(['python', 'fake_utility.py'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) t = threading.Thread(target=output_reader, args=(proc,)) t.start() try: time.sleep(0.2) import time i = 0 while True: print (hex(i)*512) i += 1 time.sleep(0.5) finally: proc.terminate() try: proc.wait(timeout=0.2) print('== subprocess exited with rc =', proc.returncode) except subprocess.TimeoutExpired: print('subprocess did not terminate in time') t.join() 

Pythont 3.5 added the methods run() and call() to the subprocess module, both returning a CompletedProcess object. With this you are fine using proc.stdout.splitlines() :

proc = subprocess.run( comman, shell=True, capture_output=True, text=True, check=True ) for line in proc.stdout.splitlines(): print "stdout:", line 

Related question

You can also read lines w/o loop. Works in python3.6.

import os import subprocess process = subprocess.Popen(command, stdout=subprocess.PIPE) list_of_byte_strings = process.stdout.readlines() 

aiven 3362

A function that allows iterating over both stdout and stderr concurrently, in realtime, line by line

In case you need to get the output stream for both stdout and stderr at the same time, you can use the following function.

The function uses Queues to merge both Popen pipes into a single iterator.

Here we create the function read_popen_pipes() :

from queue import Queue, Empty from concurrent.futures import ThreadPoolExecutor def enqueue_output(file, queue): for line in iter(file.readline, ''): queue.put(line) file.close() def read_popen_pipes(p): with ThreadPoolExecutor(2) as pool: q_stdout, q_stderr = Queue(), Queue() pool.submit(enqueue_output, p.stdout, q_stdout) pool.submit(enqueue_output, p.stderr, q_stderr) while True: if p.poll() is not None and q_stdout.empty() and q_stderr.empty(): break out_line = err_line = '' try: out_line = q_stdout.get_nowait() except Empty: pass try: err_line = q_stderr.get_nowait() except Empty: pass yield (out_line, err_line) 

import subprocess as sp with sp.Popen(my_cmd, stdout=sp.PIPE, stderr=sp.PIPE, text=True) as p: for out_line, err_line in read_popen_pipes(p): # Do stuff with each line, e.g.: print(out_line, end='') print(err_line, end='') return p.poll() # return status-code 

Rotareti 43632

You want to pass these extra parameters to subprocess.Popen :

bufsize=1, universal_newlines=True 

Then you can iterate as in your example. (Tested with Python 3.5)

Indeed, if you sorted out the iterator then buffering could now be your problem. You could tell the python in the sub-process not to buffer its output.

proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE) 

proc = subprocess.Popen(['python','-u', 'fake_utility.py'],stdout=subprocess.PIPE) 

I have needed this when calling python from within python.

Bit late to the party, but was surprised not to see what I think is the simplest solution here:

import io import subprocess proc = subprocess.Popen(["prog", "arg"], stdout=subprocess.PIPE) for line in io.TextIOWrapper(proc.stdout, encoding="utf-8"): # or another encoding # do something with line 

Related Query

• read from file line by line , multiple match in file possible
• python read file, print part of line that begins with a specific string
• How to read a specific line from a text file in python
• Read line by line and then split each object of each line — python3
• Python Subprocess — stderr erases after read
• why can’t I read the output of this subprocess call in python?
• how to read file line by line in python
• Python- how to read file line by line, between two lines with certain values
• Read Data into Python Line by Line as a List
• Popen multi line read
• How do I read a command line argument from Python?
• Wait subprocess until it outputs a certain line
• Python read line from file and sort int descending
• How does Python know that it has to read a file line by line when the sequence part of the for loop is a file?
• How can I not read the last line of a csv file if it has simply 1 column in Python?
• How to read file of subprocess output
• How to read a CSV file line by line and store it to new CSV file on new row every time?
• Python: print array of strings to stdout line by line
• Fastest way to read in very long file starting on arbitary line X
• Subprocess writing stdin and reading stdout python 3.4

More Query from same tag

• running pdftk in bash from python script
• What is the replacement of cv.Get2D in opencv 3.4?
• Why is this not a valid xpath in Python (Selenium)?
• Converting arguments to custom objects using dataclasses package
• Managing python2 and python3 at the same time
• How to detect an invalid C escaped string using a regular expression?
• Automated Test Halt with TypeError: ‘str’ object not callable
• I installed the Tensorflow gpu version and CUDA 9.1 and python 3.6.5. However, after I import the tensorflow-gpu I got the following error:
• How set a spending and description in a txt file using Python?
• Python class weird behaviour
• Loop a python script
• Keras Model Network 3D Input to 2D Output
• Tensor-Flow error: Failed to convert object of type to Tensor
• How can I create NEW Listboxes with a Button and collect all the data with a final submit Button in TKinter?
• How to return whole object on data insert instead of id in mysql flask
• How to add 2 conf in SparkContext() function?
• What is the canonical way to obtain parameters of a trained classifier in scikit-learn?
• Can’t find elements but they exist Selenium Python
• GUI Measuring Picture Size for Cropping
• Handle Thread/Greenlit kill in gevent
• Python TKinter progress bar label not acting correctly
• Escape quotes in Javascript/Chameleon template
• What is the best strategy to extract information from a JIRA issue?
• Error with String Slicing?
• TORCH ERROR: return forward_call(*input, **kwargs) AND in _forward_unimplemented raise NotImplementedError NotImplementedError
• TensorFlow text generation RNN example failing on TF 2.6, tf.sparse.to_dense(), Invalid argument: indices[1] = [0] is repeated
• how to add new key with items as value to a nested dictionary
• Can we use ternary operator inside a Python function to change no. of arguments?
• Python mock a dependancy object
• How to replace substring between two other substrings in python?
• How to fix AttributeError: ‘tuple’ object has no attribute ‘to’?
• Communicate between two scripts. Pass a string of variable from one script to another
• How to compare differently transposed data in pandas or python
• using python subprocess to a get directory size on mac instead of os.walk
• Tkinter windows floating in i3 but also working in MS-Windows

Источник