Run a Python script inside a virtual environment in the background

Published on , Updated on python High-performance computing python

Scenario

Suppose, we have a Python script that uses some Python packages. To keep the global environment clean, we use a virtual environment (venv) to install the dependencies. Let's assume, the script requires a long time to finish. We may need to check the intermediate results (e.g. print statement debugging) from the script during the execution process. We also want to run the script in the background without blocking a terminal window.

I will mock this scenario and will show how to run the Python script inside a virtual environment in the background that writes partial output to a file.

Running script in the foreground

  • Create and activate virtual environment using Python 3:

    python3 -m venv venv
    source venv/bin/activate
    
  • Suppose the required packages are listed in requirements.txt:

    numpy==1.22.3
    
  • Install the required packages:

    pip install -r requirements.txt
    
  • Create a Python script that generates 3 random integers and sleep for 5 seconds at each step. It continues the process until 30 integers are generated. The code is listed in background_script.py file:

    import time
    from datetime import datetime
    import numpy as np
        
    count = 0
    while True:
        current_time = datetime.now()
        rng = np.random.default_rng()
        random_integers = rng.integers(low=0, high=10 ** 6, size=3)
        print(f"{current_time}: {random_integers}")
        count += 1
        if count == 10:
            print(f"Script completed")
            break
        time.sleep(5)
    
  • Run the Python script (background_script.py):

    python background_script.py
    
  • It shows 3 random numbers continuously each 5 seconds. The script takes 50 seconds to complete and will stop after printing 30 random numbers:

    2022-03-18 04:18:12.376506: [874273 493185 580873]
    2022-03-18 04:18:17.378234: [175390 989369 463402]
    2022-03-18 04:18:22.382138: [721299 669516 197204]
    2022-03-18 04:18:27.386138: [537835 627442 400862]
    ...
    ...
    Script completed
    

Running script using Bash in the background

  • Let's create a Shell script (run_bg.sh) in the same directory that activates the virtual environment and runs the Python script. We have used -u parameter in the Python run command that enables printing the partial outputs to the standard output:

    #!/usr/bin/env bash
    set -e
    source "./venv/bin/activate"
    python -u background_script.py
    
  • Run the script in the foreground to check if it can activate the virtual environment and runs the Python script:

    ./run_bg.sh
    
  • If the above Shell script runs without error, we try to run it in a background process using nohup command. This will write the partial outputs to a file (custom-output.log):

    nohup ./run_bg.sh > custom-output.log &
    
  • We can see the partial output using:

    cat custom-output.log
    

    alt Run Python Script in background with partial output

  • Sometimes we may need to stop the script running in the background. We need to kill the process which runs the script to do so. The following command will list the process ID for the script:

    ps aux | grep background_script.py
    

    It will return the process details which contain the process ID, typically in the second column. Then we can kill the process by using:

    kill PROCESS_ID
    

Use case

I was trying to run a hefty Python script that requires a virtual environment in a cluster computing platform. I did not want to keep the terminal session open as the script would take days to finish. I also wanted to see the partial output while the script was running.

So, I followed this approach to run the Python script inside a virtual environment in the background that wrote the partial output to a file.

References

Advertisements

Cite This Work
APA Style
Shovon, A. R. (2022, March 18). Run a Python script inside a virtual environment in the background. Ahmedur Rahman Shovon. Retrieved March 21, 2022, from https://arshovon.com/blog/python-background/
MLA Style
Shovon, Ahmedur Rahman. “Run a Python script inside a virtual environment in the background.” Ahmedur Rahman Shovon, 18 Mar. 2022. Web. 21 Mar. 2022. https://arshovon.com/blog/python-background/.
BibTeX entry
@misc{ shovon_2022,
    author = "Shovon, Ahmedur Rahman",
    title = "Run a Python script inside a virtual environment in the background",
    year = "2022",
    url = "https://arshovon.com/blog/python-background/",
    note = "[Online; accessed 21-March-2022; URL: https://arshovon.com/blog/python-background/]"
}
Related Contents in this Website