Python run program subprocess

Running external programs with Python (os.system and subprocess)

One of the things we often have to do is glue together various programs written by other people.
If these other programs are GUI based then we will have a very hard time doing so, but if they are command line based then there are some nice ways to do that in Python. We’ll see two different ways to accomplish this.

Our external program

In order to demonstrate this we need an «external tool» that we will handle as a «black box». As you are using Python I can assume you already have Python on your computer so we’ll use a script written in Python as the «external tool». You can see it here:

import time import sys if len(sys.argv) != 3: exit(f"sys.argv[0]> SECONDS EXIT_CODE") seconds = int(sys.argv[1]) exit_code = int(sys.argv[2]) for sec in range(seconds): print("OUT <>".format(sec), flush=True) print("ERR <>".format(sec), file=sys.stderr) time.sleep(1) exit(exit_code) 
  • Output on Standard Output (STDOUT)
  • Output on Standard Error (STDERR)
  • A process that takes time
  • Various exit codes (ERRORLEVELs)

So we can see how to deal with either of those.

The user of the process.py can tell it how many iterations to do. On every iteration it will print to both STDOUT and STDERR and wait for 1 second.
The user can also tell the process how to set its exit code.

This can be a nice tool to fake the behavior of some external tool.

If we run the process as follows:

We get the following output:

OUT 0 ERR 0 OUT 1 ERR 1 OUT 2 ERR 2 

We can also observe the exit code on Linux/macOS:

Using os.system

The simplest way to run an external program is to use os.system .

import os import sys exit_code = os.system(f"python process.py 5 2") print(f'exit code: exit_code // 256>') 

It accepts a string — exactly what you would type in on the command line and executes the external program.

Your program connects its own STDOUT and STDERR channels to the external process so while the external program is running
whatever it prints to STDOUT and STDERR will be handled exactly as if they came directly from your program.

It waits till the external program ends and at the end it returns the exit-code of the external program.
Well it actually returns two bytes and the real exit code is in the higher byte so we need to do an integer division
exit_code // 256 in order to get to the real value. (It is the same as int(exit_code / 256) .)

OUT 0 ERR 0 OUT 1 ERR 1 OUT 2 ERR 2 OUT 3 ERR 3 OUT 4 ERR 4 exit code: 2 

This can be very useful, but this way the output of the external program «gets lost» to our program. Often this is not what we want.

Often we would want to capture the output of the external program, parse it and do something based on the understanding from it.

We might also want to do something else while the external program does its thing.

Let’s see how the subprocess module can help us. We will see a few examples.

subprocess waiting for external process to finish

In the first example we will imitate the os.system just to lay the ground-work.

We have created a function called run_process . Instead of a string, the command we would want to type in,
it is expected to receive a list. The pieces of the command divided up. That is probably not be a problem to write.

I sprinkled the whole program with print statements to make it easier to see what is the order of things happening.

The first thing is to call proc = subprocess.Popen(command) . This will start the external program and return immediately
passing us an object that represents this external process. ( Popen stands for process open )

At this point the external program will run regardless of what our program does. So we can wait for 1.5 seconds and see the output (and error)
of the external program. We could also do some other work while the external program runs. We’ll see that later.

At one point, however, we will likely want to wait for the external program to end. This is what the proc.communicate() does.
(It’s name is strange, I know. The next example will shed some light on why it is called that way.)
It stops our program and waits till the external program ends.

Then we can fetch the exit code (that Windows calls ERRORLEVEL) from the attribute returncode of the proc object.

(Are you already having fun by the fact that the same thing is called «exit code», ERRORLEVEL, and «returncode» by three different systems?)

import subprocess import time def run_process(command): print("Before Popen") proc = subprocess.Popen(command) # This starts runing the external process print("After Popen") time.sleep(1.5) print("Before communicate") proc.communicate() print("After communicate") exit_code = proc.returncode return exit_code print("Before run_process", flush=True) exit_code = run_process(['python', 'process.py', '5', '0']) print("After run_process", flush=True) print(f'exit code: exit_code>', flush=True) 

Источник

Subprocesses¶

This section describes high-level async/await asyncio APIs to create and manage subprocesses.

Here’s an example of how asyncio can run a shell command and obtain its result:

import asyncio async def run(cmd): proc = await asyncio.create_subprocess_shell( cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE) stdout, stderr = await proc.communicate() print(f'[cmd!r> exited with proc.returncode>]') if stdout: print(f'[stdout]\nstdout.decode()>') if stderr: print(f'[stderr]\nstderr.decode()>') asyncio.run(run('ls /zzz')) 
['ls /zzz' exited with 1] [stderr] ls: /zzz: No such file or directory 

Because all asyncio subprocess functions are asynchronous and asyncio provides many tools to work with such functions, it is easy to execute and monitor multiple subprocesses in parallel. It is indeed trivial to modify the above example to run several commands simultaneously:

async def main(): await asyncio.gather( run('ls /zzz'), run('sleep 1; echo "hello"')) asyncio.run(main()) 

Creating Subprocesses¶

coroutine asyncio. create_subprocess_exec ( program , * args , stdin = None , stdout = None , stderr = None , limit = None , ** kwds ) ¶

The limit argument sets the buffer limit for StreamReader wrappers for Process.stdout and Process.stderr (if subprocess.PIPE is passed to stdout and stderr arguments).

See the documentation of loop.subprocess_exec() for other parameters.

Changed in version 3.10: Removed the loop parameter.

coroutine asyncio. create_subprocess_shell ( cmd , stdin = None , stdout = None , stderr = None , limit = None , ** kwds ) ¶

Run the cmd shell command.

The limit argument sets the buffer limit for StreamReader wrappers for Process.stdout and Process.stderr (if subprocess.PIPE is passed to stdout and stderr arguments).

See the documentation of loop.subprocess_shell() for other parameters.

It is the application’s responsibility to ensure that all whitespace and special characters are quoted appropriately to avoid shell injection vulnerabilities. The shlex.quote() function can be used to properly escape whitespace and special shell characters in strings that are going to be used to construct shell commands.

Changed in version 3.10: Removed the loop parameter.

Subprocesses are available for Windows if a ProactorEventLoop is used. See Subprocess Support on Windows for details.

Constants¶

Can be passed to the stdin, stdout or stderr parameters.

If PIPE is passed to stdin argument, the Process.stdin attribute will point to a StreamWriter instance.

If PIPE is passed to stdout or stderr arguments, the Process.stdout and Process.stderr attributes will point to StreamReader instances.

Special value that can be used as the stderr argument and indicates that standard error should be redirected into standard output.

Special value that can be used as the stdin, stdout or stderr argument to process creation functions. It indicates that the special file os.devnull will be used for the corresponding subprocess stream.

Interacting with Subprocesses¶

Both create_subprocess_exec() and create_subprocess_shell() functions return instances of the Process class. Process is a high-level wrapper that allows communicating with subprocesses and watching for their completion.

class asyncio.subprocess. Process ¶

An object that wraps OS processes created by the create_subprocess_exec() and create_subprocess_shell() functions.

This class is designed to have a similar API to the subprocess.Popen class, but there are some notable differences:

  • unlike Popen, Process instances do not have an equivalent to the poll() method;
  • the communicate() and wait() methods don’t have a timeout parameter: use the wait_for() function;
  • the Process.wait() method is asynchronous, whereas subprocess.Popen.wait() method is implemented as a blocking busy loop;
  • the universal_newlines parameter is not supported.

Wait for the child process to terminate.

Set and return the returncode attribute.

This method can deadlock when using stdout=PIPE or stderr=PIPE and the child process generates so much output that it blocks waiting for the OS pipe buffer to accept more data. Use the communicate() method when using pipes to avoid this condition.

  1. send data to stdin (if input is not None );
  2. read data from stdout and stderr, until EOF is reached;
  3. wait for process to terminate.

The optional input argument is the data ( bytes object) that will be sent to the child process.

Return a tuple (stdout_data, stderr_data) .

If either BrokenPipeError or ConnectionResetError exception is raised when writing input into stdin, the exception is ignored. This condition occurs when the process exits before all data are written into stdin.

If it is desired to send data to the process’ stdin, the process needs to be created with stdin=PIPE . Similarly, to get anything other than None in the result tuple, the process has to be created with stdout=PIPE and/or stderr=PIPE arguments.

Note, that the data read is buffered in memory, so do not use this method if the data size is large or unlimited.

Sends the signal signal to the child process.

On Windows, SIGTERM is an alias for terminate() . CTRL_C_EVENT and CTRL_BREAK_EVENT can be sent to processes started with a creationflags parameter which includes CREATE_NEW_PROCESS_GROUP .

On POSIX systems this method sends signal.SIGTERM to the child process.

On Windows the Win32 API function TerminateProcess() is called to stop the child process.

On POSIX systems this method sends SIGKILL to the child process.

On Windows this method is an alias for terminate() .

Standard input stream ( StreamWriter ) or None if the process was created with stdin=None .

Standard output stream ( StreamReader ) or None if the process was created with stdout=None .

Standard error stream ( StreamReader ) or None if the process was created with stderr=None .

Use the communicate() method rather than process.stdin.write() , await process.stdout.read() or await process.stderr.read() . This avoids deadlocks due to streams pausing reading or writing and blocking the child process.

Process identification number (PID).

Note that for processes created by the create_subprocess_shell() function, this attribute is the PID of the spawned shell.

Return code of the process when it exits.

A None value indicates that the process has not terminated yet.

A negative value -N indicates that the child was terminated by signal N (POSIX only).

Subprocess and Threads¶

Standard asyncio event loop supports running subprocesses from different threads by default.

On Windows subprocesses are provided by ProactorEventLoop only (default), SelectorEventLoop has no subprocess support.

On UNIX child watchers are used for subprocess finish waiting, see Process Watchers for more info.

Changed in version 3.8: UNIX switched to use ThreadedChildWatcher for spawning subprocesses from different threads without any limitation.

Spawning a subprocess with inactive current child watcher raises RuntimeError .

Note that alternative event loop implementations might have own limitations; please refer to their documentation.

Examples¶

An example using the Process class to control a subprocess and the StreamReader class to read from its standard output.

The subprocess is created by the create_subprocess_exec() function:

import asyncio import sys async def get_date(): code = 'import datetime; print(datetime.datetime.now())' # Create the subprocess; redirect the standard output # into a pipe. proc = await asyncio.create_subprocess_exec( sys.executable, '-c', code, stdout=asyncio.subprocess.PIPE) # Read one line of output. data = await proc.stdout.readline() line = data.decode('ascii').rstrip() # Wait for the subprocess exit. await proc.wait() return line date = asyncio.run(get_date()) print(f"Current date: date>") 

See also the same example written using low-level APIs.

Источник

Читайте также:  Питон тьютор ответы цикл while
Оцените статью