Run a Python script inside a virtual environment in the background

Scenario

Suppose, we have a Python script that uses some Python packages. To keep the global environment clean, we use a virtual environment (venv) to install the dependencies. Let's assume, the script requires a long time to finish. We may need to check the intermediate results (e.g. print statement debugging) from the script during the execution process. We also want to run the script in the background without blocking a terminal window.

I will mock this scenario and will show how to run the Python script inside a virtual environment in the background that writes partial output to a file.

Running script in the foreground

• Create and activate virtual environment using Python 3:

  python3 -m venv venv
  source venv/bin/activate
  

• Suppose the required packages are listed in requirements.txt:

  numpy==1.22.3
  

• Install the required packages:

  pip install -r requirements.txt
  

• Create a Python script that generates 3 random integers and sleep for 5 seconds at each step. It continues the process until 30 integers are generated. The code is listed in background_script.py file:

  import time
  from datetime import datetime
  import numpy as np
      
  count = 0
  while True:
      current_time = datetime.now()
      rng = np.random.default_rng()
      random_integers = rng.integers(low=0, high=10 ** 6, size=3)
      print(f"{current_time}: {random_integers}")
      count += 1
      if count == 10:
          print(f"Script completed")
          break
      time.sleep(5)
  

• Run the Python script (background_script.py):

  python background_script.py
  

• It shows 3 random numbers continuously each 5 seconds. The script takes 50 seconds to complete and will stop after printing 30 random numbers:

  2022-03-18 04:18:12.376506: [874273 493185 580873]
  2022-03-18 04:18:17.378234: [175390 989369 463402]
  2022-03-18 04:18:22.382138: [721299 669516 197204]
  2022-03-18 04:18:27.386138: [537835 627442 400862]
  ...
  ...
  Script completed
  

Running script using Bash in the background

• Let's create a Shell script (run_bg.sh) in the same directory that activates the virtual environment and runs the Python script. We have used -u parameter in the Python run command that enables printing the partial outputs to the standard output:

  #!/usr/bin/env bash
  set -e
  source "./venv/bin/activate"
  python -u background_script.py
  

• Run the script in the foreground to check if it can activate the virtual environment and runs the Python script:

  ./run_bg.sh
  

• If the above Shell script runs without error, we try to run it in a background process using nohup command. This will write the partial outputs to a file (custom-output.log):

  nohup ./run_bg.sh > custom-output.log &
  

• We can see the partial output using:

  cat custom-output.log
  

  

• Sometimes we may need to stop the script running in the background. We need to kill the process which runs the script to do so. The following command will list the process ID for the script:

  ps aux | grep background_script.py
  

  It will return the process details which contain the process ID, typically in the second column. Then we can kill the process by using:

  kill PROCESS_ID
  

Use case

I was trying to run a hefty Python script that requires a virtual environment in a cluster computing platform. I did not want to keep the terminal session open as the script would take days to finish. I also wanted to see the partial output while the script was running.

So, I followed this approach to run the Python script inside a virtual environment in the background that wrote the partial output to a file.

References

• sh command man page
• nohup(1) — Linux manual page