Shell command in Python

How to Execute a Shell Command in Python [Step-by-Step]

by

Knowing how to execute a shell command in Python helps you create programs to automate tasks on your system.

There are multiple ways to execute a shell command in Python. The simplest ones use the os.system and os.popen functions. The recommended module to run shell commands is the Python subprocess module due to its flexibility in giving you access to standard output, standard error and command piping.

We will start this tutorial with the os module and then we will move to the subprocess module.

This will give you a full understanding of how to handle shell commands in Python.

Let’s start coding!

Using OS System to Run a Command in Python

I have created a simple Python script called shell_command.py.

It uses the system function of the os module to run the Linux date command:

import os

os.system('date')

This is the output of the os.system() function:

$ python shell_command.py
Sun Feb 21 16:01:43 GMT 2021

Let’s see what happens when we run the same code in the Python shell:

>>> import os
>>> os.system('date')
Sun Feb 21 16:01:43 GMT 2021
0

We still see the output of the date command but we also see a 0 in the last line. That’s the exit code of the Linux command.

A successful command in Linux returns a 0 exit code and a non-zero exit code is returned in case of failure.

Let’s confirm that, by introducing a spelling mistake in the date command:

>>> os.system('daet')
>>> sh: daet: command not found
>>> 32512

Notice how the exit status is different from the one returned by the Bash shell:

$ daet
-bash: daet: command not found
$ echo $?
127

I have written another article that explains Bash exit codes if it’s something you would like to know more about.

Later on in this article we will make a comparison between os.system and a different Python module called subprocess.

Using OS Popen to Execute Commands

Using os.system() we cannot store the output of the Linux command into a variable. And this is one of the most useful things to do when you write a script.

You usually run commands, store their output in a variable and then implement some logic in your script that does whatever you need to do with that variable (e.g. filter it based on certain criteria).

To be able to store the output of a command in a variable you can use the os.popen() function.

The popen function returns an open file object and to read its value you can use the read method:

>>> import os
>>> output = os.popen('date')
>>> type(output)
<class 'os._wrap_close'>
>>> print(output.__dict__)
{'_stream': <_io.TextIOWrapper name=3 encoding='UTF-8'>, '_proc': }         
>>> output.read()
'Sun Feb 21 16:01:43 GMT 2021\n'

Have a look at what happens if we use the read method again on the same object:

>>> output.read()
''

The result is an empty string because we can only read from the file object once.

We can use os.popen and the read function in a single line:

>>> import os
>>> print(os.popen('date').read())
Sun Feb 21 16:01:43 GMT 2021

When executing shell commands in Python it’s important to understand if a command is executed successfully or not.

To do that we can use the close method of the file object returned by os.popen. The close method returns None if the command is executed successfully. It provides the return code of the subprocess in case of error.

Success scenario

>>> output = os.popen('date')
>>> print(output.close())
None

Failure scenario

>>> output = os.popen('daet')
>>> /bin/sh: daet: command not found

>>> print(output.close())
32512

We can use the value of output.close() for error handling in our Python scripts.

Makes sense?

Do OS System and OS Popen Wait for Command Completion?

Before moving to a different way of running shell commands in Python, I want to see the behaviour of os.system() and os.popen() with a command that takes few seconds to complete.

We will use the ping command with the -c flag to stop the execution of the command after a specific number of ECHO_RESPONSE packets (in this example 5):

$ ping -c 5 localhost

Execution with os.system

>>> os.system('ping -c 5 localhost')
PING localhost (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.051 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.068 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.091 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.066 ms
64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.063 ms

--- localhost ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.051/0.068/0.091/0.013 ms
0

When executing the command with os.system() we see the output for each ping attempt being printed one at the time, in the same way we would see it in the Linux shell.

Execution with os.popen

>>> os.popen('ping -c 5 localhost')
<os._wrap_close object at 0x10bc8a190> 
>>> os.popen('ping -c 5 localhost').read()
'PING localhost (127.0.0.1): 56 data bytes\n64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.055 ms\n64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.059 ms\n64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.073 ms\n64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.135 ms\n64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.077 ms\n\n--- localhost ping statistics ---\n5 packets transmitted, 5 packets received, 0.0% packet loss\nround-trip min/avg/max/stddev = 0.055/0.080/0.135/0.029 ms\n'

When using os.popen() we don’t see the output in the Python shell immediately.

The output gets buffered, we only see the final output once the ping command is complete.

The os.popen function waits for the completion of a command before providing the full output.

Very soon we will look at the difference between the Popen function of the OS module and the Popen function of the subprocess module.

Read, Readline and Readlines Methods Applied to the OS.Popen Output

We have seen that:

  • os.popen() returns an open file object.
  • we can read the content of the object using the read() method.

In this section we will compare the behavior of the read(), readline() and readlines() methods applied to the file object returned by os.popen.

We know already that the read method returns the output of the command as soon as the command execution is complete.

Let’s see what happens with the readline method:

>>> output = os.popen('ping -c 5 localhost')
>>> output.readline()
'PING localhost (127.0.0.1): 56 data bytes\n'
>>> output.readline()
'64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.047 ms\n'
>>> output.readline()
'64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.067 ms\n'
>>> output.readline()
'64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.055 ms\n'
>>> output.readline()
'64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.103 ms\n'
>>> output.readline()
'64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.058 ms\n'
>>> output.readline()
'\n'
>>> output.readline()
'--- localhost ping statistics ---\n'
>>> output.readline()
'5 packets transmitted, 5 packets received, 0.0% packet loss\n'
>>> output.readline()
'round-trip min/avg/max/stddev = 0.047/0.066/0.103/0.020 ms\n'
>>> output.readline()
''
>>> output.readline()
''

With the readline method we can print the output of the command one line at the time until we reach the end of the open file object.

Here is how you can use the readline method with a while loop, to print the full output of the command:

import os

output = os.popen('ping -c 5 localhost')

while True:
    line = output.readline()
 
    if line:
        print(line, end='')
    else:
        break

 output.close()

On the other side, the readlines method waits for the command to complete and it returns a Python list:

>>> output = os.popen('ping -c 5 localhost')
>>> output.readlines()
['PING localhost (127.0.0.1): 56 data bytes\n', '64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.044 ms\n', '64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.095 ms\n', '64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.057 ms\n', '64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.078 ms\n', '64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.094 ms\n', '\n', '--- localhost ping statistics ---\n', '5 packets transmitted, 5 packets received, 0.0% packet loss\n', 'round-trip min/avg/max/stddev = 0.044/0.074/0.095/0.020 ms\n']

And with a simple for loop we can print the full output of the command by going through all the elements in the list returned by the readlines() method:

import os

output = os.popen('ping -c 5 localhost')

for line in output.readlines():
    print(line, end='')

All clear? 🙂

Run Shell Commands in Python with Subprocess.Run

In the previous section we have seen how to run the date command using os.system and os.popen.

Now, you will learn how to use the subprocess module to run the same command.

There are few options to run commands using subprocess and I will start with the recommended one if you are using Python 3.5 or later: subprocess.run.

Let’s pass the date command to subprocess.run():

>>> import subprocess
>>> subprocess.run('date')
Sun Feb 21 21:44:53 GMT 2021
CompletedProcess(args='date', returncode=0)

As you can see, the command returns an object of type CompletedProcess (it’s actually subprocess.CompletedProcess).

Let’s see what happens if I also pass the +%a flag to the date command (it should show the day of the week):

import subprocess
   
subprocess.run('date +%a')

We see the following error:

$ python subprocess_example.py 
Traceback (most recent call last):
  File "subprocess_example.py", line 3, in <module>
    subprocess.run('date +%a')
  File "/Users/codefather/opt/anaconda3/lib/python3.7/subprocess.py", line 472, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/Users/codefather/opt/anaconda3/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/Users/codefather/opt/anaconda3/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'date +%a': 'date +%a'

One way to make this command work is by passing the shell=True parameter to subprocess.run():

import subprocess
   
subprocess.run('date +%a', shell=True)

Give it a try and confirm that the command works as epxected.

Passing the parameter shell=True the command gets invoked through the shell.

Note: Be aware of security considerations related to the use of the shell parameter.

When you execute the command, the run() function waits for the command to complete. I will explain you more about this in one of the next sections.

If we want to run the “date +%a” command without passing shell=True we have to pass date and its flags as separate elements of an array.

import subprocess
   
subprocess.run(['date', '+%a'])

[output]
Sun
CompletedProcess(args=['date', '+%a'], returncode=0)

Capture the Command Output with Subprocess.run

So far we have printed the output of the command in the shell.

But, what if we want to store the output of the command in a variable?

Could we simply add a variable to the left side of the subprocess.run() call?

Let’s find out…

import subprocess

process_output = subprocess.run(['date', '+%a'])
print(process_output)

[output]
Sun
CompletedProcess(args=['date', '+%a'], returncode=0)

We still see the output of the command in the shell and the print statement shows that the variable process_output is a CompletedProcess object.

Let’s find out the attributes of the object…

To see the namespace associated to this Python object we can use the __dict__ method.

print(process_output.__dict__)

[output]
{'args': ['date', '+%a'], 'returncode': 0, 'stdout': None, 'stderr': None}

You can see the attributes in which args, return code, standard output and standard error are stored.

The return code for the last command we have executed is zero. Once again. the return code for a successful command is zero.

Let’s see what the return code is if we introduce a syntax error in the command:

process_output = subprocess.run(['date', '%a'])

This is the error we see after passing an incorrect flag to the date command:

$ python subprocess_example.py 
date: illegal time format
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] …
            [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
{'args': ['date', '%a'], 'returncode': 1, 'stdout': None, 'stderr': None}

The return code is 1 (non-zero return codes indicate failure).

Also, the stdout is None because the output of the command is sent to the terminal.

How can we capture the stdout in a variable?

In the official subprocess documentation I can see the following:

shell command in Python - subprocess.run

So, let’s find out what happens if we set capture_output to True…

process_output = subprocess.run(['date', '+%a'], capture_output=True)

Run this command, you will not see the output of the command printed in the shell unless you use a print statement to show the value of the variable process_output:

>>> import subprocess
>>> process_output = subprocess.run(['date', '+%a'], capture_output=True)
>>> print(process_output)
CompletedProcess(args=['date', '+%a'], returncode=0, stdout=b'Sun\n', stderr=b'')

This time you can see that the value of stdout and stderr is not None anymore:

  • The standard output contains the output of the command.
  • The standard error is an empty string because the date command has been executed successfully.

Let’s also confirm that stderr is not empty in case of error:

>>> import subprocess
>>> process_output = subprocess.run(['date', '%a'], capture_output=True)
>>> print(process_output)
CompletedProcess(args=['date', '%a'], returncode=1, stdout=b'', stderr=b'date: illegal time format\nusage: date [-jnRu] [-d dst] [-r seconds] [-t west] [ v[+|-]val[ymwdHMS]] ... \n            [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]\n')

The error is stored in process_output.stderr.

The Format of Standard Output and Standard Error of a Command

In the last command we have seen that stdout and stderr have a format that is not very readable.

That’s because they are both captured as bytes (notice all the new line characters in the values of stdout and stderr).

What if we want to see them in the same format of the output of a command in the shell?

We can pass the additional parameter text to the subprocess.run method:

>>> import subprocess
>>> process_output = subprocess.run(['date', '+%a'], capture_output=True, text=True)
>>> print(process_output.stdout)
Sun

Note: the text parameter was introduced in Python 3.7 as a more understandable alternative to the parameter universal_newlines.

As an alternative you can also convert the bytes into a string using the decode method:

>>> import subprocess
>>> process_output = subprocess.run(['date', '+%a'], capture_output=True)
>>> print(process_output.stdout)
b'Sun\n' 
>>> print(process_output.stdout.decode())
Sun

Do you see the difference in the format of the stdout with and without decode?

Before, in the definition of the capture_output parameter we have seen that passing it, it’s the same as passing stdout=PIPE and stderr=PIPE.

Let’s try to use those instead to confirm the output is the same…

>>> import subprocess
>>> process_output = subprocess.run(['date', '+%a'], stdout=PIPE, stderr=PIPE, text=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'PIPE' is not defined

We have got the error “name ‘PIPE’ is not defined“. Why?

As you can see from the definition below, coming from the official subprocess documentation, PIPE is part of the subprocess module. This mean we have to use subprocess.PIPE in our program.

shell command in Python - subprocess.pipe
>>> import subprocess
>>> process_output = subprocess.run(['date', '+%a'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
>>> print(process_output.stdout)
Sun

Looks better now 🙂

How to Capture Standard Output and Standard Error in One Single Stream

To capture standard output and standard error in a single stream we have to set stdout to subprocess.PIPE and stderr to subprocess.STDOUT:

>>> process_output = subprocess.run(['date', '+%a'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)

The output is:

>> print(process_output.__dict__)
{'args': ['date', '+%a'], 'returncode': 0, 'stdout': 'Sun\n', 'stderr': None}

The stdout contains the output and the value of the stderr is None.

And, what if there is an error in the execution of the command?

>>> process_output = subprocess.run(['date', '%a'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
>>> print(process_output.__dict__)
{'args': ['date', '%a'], 'returncode': 1, 'stdout': 'date: illegal time format\nusage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] … \n            [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]\n', 'stderr': None}

As expected the error is part of the stdout stream. The value of the stderr attribute is still None.

Writing the Output of a Command to a File in Python

You can also write the output of a command to a file.

Let’s see how using the Python with statement…

with open('command.out', 'w') as stdout_file:
    process_output = subprocess.run(['date', '+%a'], stdout=stdout_file, stderr=subprocess.PIPE, text=True)
    print(process_output.__dict__)

Notice how this time the value of stdout is None considering that we are sending the stdout to a file.

$ python subprocess_example.py 
{'args': ['date', '+%a'], 'returncode': 0, 'stdout': None, 'stderr': ''}
$ ls -ltr
total 16
-rw-r--r--  1 myuser  mygroup  208 Feb 21 23:45 subprocess_example.py
-rw-r--r--  1 myuser  mygroup    4 Feb 21 23:46 command.out
$ cat command.out
Sun

Using the ls and cat commands we confirm that the command.out file has been created and that it contains the output of the command executed in our Python program.

What about writing also the standard error to a file?

To do that we can open two files using the Python with statement.

with open('command.out', 'w') as stdout_file, open('command.err', 'w') as stderr_file:
    process_output = subprocess.run(['date', '+%a'], stdout=stdout_file, stderr=stderr_file, text=True)
    print(process_output.__dict__)

This time both stdout and stderr are set to None and the two files gets created when we invoke the command (the command.err file is empty because the command is executed successfully).

$ python subprocess_example.py
{'args': ['date', '+%a'], 'returncode': 0, 'stdout': None, 'stderr': None}
$ ls -ltr
total 16
-rw-r--r--  1 myuser  mygroup  245 Feb 21 23:53 subprocess_example.py
-rw-r--r--  1 myuser  mygroup    0 Feb 21 23:55 command.err
-rw-r--r--  1 myuser  mygroup    4 Feb 21 23:55 command.out

Before continuing try to exeucte a command with an incorrect syntax and confirm that the error is written to the command.err file.

Redirect Command Output to /dev/null

You might have the requirement to redirect the command output to /dev/null.

To do that we can use a special value provided by the subprocess module: DEVNULL.

process_output = subprocess.run(['date', '%a'], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT, text=True)
print(process_output)

The object returned by subprocess.run doesn’t include the stdout and stderr attributes:

$ python subprocess_example.py
CompletedProcess(args=['date', '%a'], returncode=1)

How to Throw a Python Exception when a Shell Command Fails

In one of the previous examples we have seen what happens if we run a command with incorrect syntax:

>>> process_output = subprocess.run(['date', '%a'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)

The error gets stored in the stderr stream but Python doesn’t raise any exceptions.

To be able to catch these errors we can pass the parameter check=True to subprocess.run.

>>> process_output = subprocess.run(['date', '%a'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, check=True)
Traceback (most recent call last):
  File "", line 1, in 
  File "/Users/codefather/opt/anaconda3/lib/python3.7/subprocess.py", line 487, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['date', '%a']' returned non-zero exit status 1.

Python raises a subprocess.CalledProcessError exception that we can catch as part of a try and except block.

import subprocess

try:
    process_output = subprocess.run(['date', '%a'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, check=True)
except subprocess.CalledProcessError:
    print("Error detected while executing the command")

Now we can handle errors in a better way:

$ python subprocess_example.py
Error detected while executing the command

Nice one 🙂

How to Run Multiple Commands with Subprocess

It’s very common in Linux to use the pipe to send the output of a command as input of another command.

We will see how it’s possible to do the same in Python with subprocess.

We will execute the first command in the same way we have done before and then execute a second command that receives the additional paramer input.

The value of input will be set to the stdout of the first command.

It’s easier to show it with an example…

I have created a file that has six lines:

$ cat test_file
line1
line2
line3
line4
line5
line6

And I want to run the following command in Python:

$ wc -l test_file | awk '{print $1}'
6

We will have to take the output of the wc command and pass it as input of the awk command.

import subprocess

wc_cmd = subprocess.run(['wc', '-l', 'test_file'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print("wc_cmd object: {}".format(wc_cmd.__dict__))

awk_cmd = subprocess.run(['awk', '{print $1}'], input=wc_cmd.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print("awk_cmd object: {}".format(awk_cmd.__dict__))

print("The ouput of the command is: {}".format(awk_cmd.stdout.decode()))

You can see the two commands executed using subprocess.run.

We are also passing the input parameter to the second command (awk) execution and its value is set to the stdout of the first command (wc).

The final output is:

$ python subprocess_example.py
wc_cmd object: {'args': ['wc', '-l', 'test_file'], 'returncode': 0, 'stdout': b'       6 test_file\n', 'stderr': b''}
awk_cmd object: {'args': ['awk', '{print $1}'], 'returncode': 0, 'stdout': b'6\n', 'stderr': b''}
The ouput of the command is: 6

We could also execute the command with a single subprocess.run call passing shell=True:

>>> import subprocess
>>> wc_awk_cmd = subprocess.run("wc -l test_file | awk '{print $1}'", shell=True)
6

Shlex.split and the Subprocess Module

So far, we have seen that to run a command using subprocess.run we have to pass a list where the first element is the command and the other elements are flags you would normally pass in the shell separated by space.

For a long command it can be tedious having to create this list manually. A solution for this is the shlex module, specifically the split function.

Let’s take as an example the wc command we have used in the previous section:

wc -l test_file

Here is what happens when you apply shlex.split to this string:

>>> import shlex
>>> shlex.split('wc -l test_file')
['wc', '-l', 'test_file']

That’s exactly the format of the argument we need to pass to subprocess.run.

It’s time to run our command using shlex.split:

import subprocess, shlex
   
cmd = 'wc -l test_file'
wc_cmd = subprocess.run(shlex.split(cmd), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print(wc_cmd.__dict__)

[output]
{'args': ['wc', '-l', 'test_file'], 'returncode': 0, 'stdout': b'       6 test_file\n', 'stderr': b''}

Print Shell Environment Variables in Python

You might want to use shell environment variables in your Python program.

Let’s find out how to do that…

>>> import subprocess
>>> echo_cmd = subprocess.run(['echo', '$SHELL'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> print(echo_cmd.__dict__)
{'args': ['echo', '$SHELL'], 'returncode': 0, 'stdout': b'$SHELL\n', 'stderr': b''}

When I try to execute “echo $SHELL” using subprocess.run the stdout is simply the string $SHELL.

Our program is not resolving the value of the environment variable $SHELL. To do that we have to use os.path.expandvars(“$SHELL”).

>>> import os
>>> echo_cmd = subprocess.run(['echo', os.path.expandvars("$SHELL")], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> print(echo_cmd.__dict__)
{'args': ['echo', '/bin/bash'], 'returncode': 0, 'stdout': b'/bin/bash\n', 'stderr': b''}

Using Subprocess with SSH

You can also use subprocess to execute commands on a remote system via SSH.

Here’s how:

import subprocess, shlex
   
cmd = "ssh -i ~/.ssh/id_rsa youruser@yourhost"
ssh_cmd = subprocess.Popen(shlex.split(cmd), stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
ssh_cmd.stdin.write("date")
ssh_cmd.stdin.close()
print(ssh_cmd.stdout.read())

Have a look at how we use the standard input to invoke the date command via SSH.

And here is the script output:

$ python subprocess_example.py 
Mon 22 Feb 11:58:50 UTC 2021

Subprocess.run vs Subprocess.call

In versions of Python before 3.5 subprocess.run() is not present. You can use subprocess.call() instead.

Here is what the official documentation says about the call function…

shell command in Python - subprocess.call

Let’s use subprocess.call to run the ping command we have seen before in this tutorial. For a moment I’m assuming that I can run the command using subprocess.call using the same syntax as subprocess.run.

Let’s see if it’s true…

import subprocess, shlex
   
cmd = 'ping -c 5 localhost'
ping_cmd = subprocess.call(shlex.split(cmd), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print(ping_cmd)

But I get an error back:

$ python subprocess_example.py 
Traceback (most recent call last):
  File "subprocess_example.py", line 5, in <module>
    print(ping_cmd.__dict__)
AttributeError: 'int' object has no attribute '__dict__'

That’s because from subprocess.call we don’t get back an object but just the integer for the return code:

>>> import subprocess, shlex
>>> cmd = 'ping -c 5 localhost'
>>> ping_cmd = subprocess.call(shlex.split(cmd), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> print(ping_cmd)
0

Looking at the subprocess.call documentation I notice the following message:

Python subprocess - stdout=PIPE or stderr=PIPE

How do we retrieve the output of subprocess.call then?

The official documentation suggests that if you need to capture stdout or stderr you should use subprocess.run() instead.

Before closing this section I wanted to give a quick try to subprocess.check_call that is also present in the documentation.

But then I realised that also in this case the documentation suggests to use run().

Subprocess.run vs Subprocess.Popen

In the last section of this tutorial we will test an alternative to subprocess.run: subprocess.Popen.

The reason I want to show you this it’s because there is something quite interesting in the behaviour of subprocess.Popen.

Let’s start by running the ping command we have used for other examples before, using subprocess.run:

>>> import subprocess, shlex
>>> cmd = 'ping -c 5 localhost'
>>> ping_cmd = subprocess.run(shlex.split(cmd), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> print(ping_cmd.__dict__)
{'args': ['ping', '-c', '5', 'localhost'], 'returncode': 0, 'stdout': b'PING localhost (127.0.0.1): 56 data bytes\n64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.075 ms\n64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.056 ms\n64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.158 ms\n64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.065 ms\n64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.074 ms\n\n--- localhost ping statistics ---\n5 packets transmitted, 5 packets received, 0.0% packet loss\nround-trip min/avg/max/stddev = 0.056/0.086/0.158/0.037 ms\n', 'stderr': b''}

Subprocess.run waits for the command to complete when you press ENTER to execute the line calling subprocess.run in the Python shell (I suggest to run this on your machine to see this behaviour).

Now, let’s run the same command using subprocess.Popen…

>>> ping_cmd = subprocess.Popen(shlex.split(cmd), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> print(ping_cmd.__dict__)
{'_waitpid_lock': <unlocked _thread.lock object at 0x102e60f30>, '_input': None, '_communication_started': False, 'args': ['ping', '-c', '5', 'localhost'], 'stdin': None, 'stdout': <_io.BufferedReader name=3>, 'stderr': <_io.BufferedReader name=5>, 'pid': 35340, 'returncode': None, 'encoding': None, 'errors': None, 'text_mode': None, '_sigint_wait_secs': 0.25, '_closed_child_pipe_fds': True, '_child_created': True}

Subprocess.Popen returns immediately when you press ENTER in the Python shell. Also the object returned is very different from the subprocess.run one.

To get standard output and standard error we have to use the communicate() function that returns a tuple in which the first element is the stdout and the second element is the stderr.

>>> stdout, stderr = ping_cmd.communicate()
>>> print(stdout)
b'PING localhost (127.0.0.1): 56 data bytes\n64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.060 ms\n64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.061 ms\n64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.059 ms\n64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.103 ms\n64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.119 ms\n\n--- localhost ping statistics ---\n5 packets transmitted, 5 packets received, 0.0% packet loss\nround-trip min/avg/max/stddev = 0.059/0.080/0.119/0.025 ms\n'
>>> print(stderr)
b''

Let’s go back to the fact that subprocess.Popen was executed immediately (in a non-blocking way) in the Python shell even if the ping command did not complete immediately.

With subprocess.Popen we can poll the status of a long running command, here’s how:

import subprocess, shlex, time
   
cmd = 'ping -c 5 localhost'
ping_cmd = subprocess.Popen(shlex.split(cmd), stdout=subprocess.PIPE, stderr=subprocess.PIPE) 

while True:
    return_code = ping_cmd.poll()
    print("Return code: {}".format(return_code))

    if return_code is not None:
        break
    else:
        time.sleep(1)
        print("Command in progress...\n") 

print("Command completed with return code: {}".format(return_code))
print("Command output: {}".format(ping_cmd.stdout.read()))

The poll() function returns None while the command is executed.

We can use this to exit from the while loop only when the execution of the command is complete based on the fact that the return code is not None. 

$ python subprocess_example.py 
Return code: None
Command in progress... 

Return code: None
Command in progress... 

Return code: None
Command in progress... 

Return code: None
Command in progress...

Return code: None
Command in progress... 

Return code: 0
Command completed with return code: 0
Command output: b'PING localhost (127.0.0.1): 56 data bytes\n64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.068 ms\n64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.066 ms\n64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.088 ms\n64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.095 ms\n64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.071 ms\n\n--- localhost ping statistics ---\n5 packets transmitted, 5 packets received, 0.0% packet loss\nround-trip min/avg/max/stddev = 0.066/0.078/0.095/0.012 ms\n'

Makes sense?

Conclusion

We have seen lots of different ways to execute a shell command in Python.

Well done for completing this tutorial!

The recommended way to invoke shell commands is definitely subprocess.run unless you are not using Python 3.5+. In that case you can use subprocess.Popen.

It takes practice to get used to the syntax of the subprocess module, so make sure you try the examples we have covered in this tutorial on your own machine.

Happy coding! 😀

Related posts:

  1. Ternary Operator in Python: To Use or Not to Use?
  2. Python Yield: Create Your Generators [With Examples]
  3. How to Create a Stack in Python From Scratch: Step-By-Step
  4. How to Implement the Binary Search Algorithm in Python

1 thought on “How to Execute a Shell Command in Python [Step-by-Step]”

Leave a Comment