Skip to content

Python Virtual Environments

This page covers installing your own version of Python 3. These are called “Python virtual environments” and are cloned from the HPC’s system Python into your own home directory. They are about 250 MB in size. This is much smaller than a Miniconda install. See the page Install Miniconda Python.

Why should you use virtual Python environments?
Your own version of Python would be needed if your application or code requires additional Python packages, or a specific version of Python or Python packages. Importantly it also means that you control what versions of Python packages are installed and they will only be upgraded if you do so. Our system Python upgrades will not affect your Python virtual environment.

We will be using the command virtualenv. See virtualenv --help

What is the HPC’s System Python?

The HPC’s system Python is the one that would be invoked if you just typed python2 or python3.

hpcnode01 $ which python2

hpcnode01 $ which python3

hpcnode01 $ python2 --version
Python 2.7.17

hpcnode01 $ python3 --version
Python 3.6.8

hpcnode01 $ python3.6 --version
Python 3.6.8

hpcnode01 $ python3.8 --version
Python 3.8.17

Any of these environments can be cloned to create your own Python environment based off the HPC’s Python system.

Installing your own Python 3.x Environment

There are two Python 3 versions on the cluster: /usr/bin/python3.6 and /usr/bin/python3.8. You can use either one. Let’s create a new Python 3.8 environment based on /usr/bin/python3.8.

I store all my Python virtual environments in a subdirectory “virtualenvs” so I change to that directory first.

$ cd virtualenvs

The command below is explained here:
we invoke the Python executable that we wish to clone i.e. /usr/bin/python3.8
we also need to load its virtual environment module i.e. -m venv
and we tell it where to create that cloned environment i.e. into a directory test_py38.
Optionally we can also tell it to use a more descriptive prompt when that environment is loaded i.e. “Test Py3.8”.

$ /usr/bin/python3.8 -m venv test_py38 --prompt="Test Py3.8"
$ source test_py38/bin/activate
$ python -m pip install --upgrade pip

Note: for the directory where the environment will be created do not use any spaces i.e. test_py38, however for the nicer prompt you can use spaces i.e. “Test Py3.8”.

You will probably prefer to use a directory name for your Python environment based on whatever project you are working on rather than test_py38.

Now you need to “activate” this Python environment. Note: you must use source test_py38/bin/activate, you cannot run activate directly like test_py38/bin/activate.

virtualenvs$ source test_py38/bin/activate 
(Test Py38) ~/virtualenvs$ 

The shell prompt will now show the name of the Python environment currently active i.e. (Test Py38). The path to this new Python i.e. ~/test_py38/bin will have been added to the start of your existing PATH. It will have also added a “VIRTUAL_ENV” environment variable.

(Test Py38) $ which python

(Test Py38) $ python -V
Python 3.8.12

(test_py38) $ echo $VIRTUAL_ENV

It’s probably a good idea at this stage to make sure that Python’s package manager “pip” is up-to-date:

(Test Py38) $ python -m pip install --upgrade pip

To exit this Python environment use “deactivate”. This will return you to the default system Python environment.

(Test Py38) ~$ deactivate

Using a Python Virtualenv with PBS

Within a PBS submission script you will need to activate your Python virtual environment, just the same as you would do at the command line. Add the line, like the one below, before the line which runs your python code.

$ source $HOME/virtualenvs/test_py38/bin/activate

and at the end of your PBS submission script, in the “Cleanup” section add:

$ deactivate

Those are the only changes required.

Upgrade pip and Other Packages

The first thing you should do when you create a Python virtual environment is to upgrade its Python package manager “pip”.

virtualenvs/$ source test_py38/bin/activate
(test Py38)$ 

(Test Py38)$ python -m pip install --upgrade pip
You are using pip version 7.1.0, however version 19.1.1 is available.
Found existing installation: pip 7.1.0
Uninstalling pip-7.1.0:
Successfully uninstalled pip-7.1.0
Successfully installed pip-19.1.1
(Test Py38)$ 

Now you should consider upgrading all the currently installed packages. You can get a list of Python packages which are outdated with:

(Test Py38) $ pip list --outdated
Package     Version  Latest 
----------  -------  ------ 
h5py        2.8.0    2.9.0  
numpy       1.15.1   1.16.2 
pip         18.0     19.0.3 
scipy       1.1.0    1.2.1  
setuptools  12.0.5   40.8.0 
six         1.11.0   1.12.0 

You can upgrade a package like “numpy” like this:

$ pip install --upgrade numpy

Another way to do this is to install the pip-review package and use that:

$ pip install pip-review
$ pip-review --interactive

The interactive option will ask you if you wish to upgrade for each package or all the packages at once. You can also use the auto option. See pip-review -h.

Create a requirements.txt File

It’s really important to record what packages, and at what versions, are installed in each virtual environment. Running pip freeze will output a list of each installed package and its version. This output can then be used to reinstall the packages again into this virtual environment with pip install -r requirements.txt.

Create your “requirements.txt” file:

virtualenvs$ source test_py38/bin/activate 
(Test Py38) ~/virtualenvs$

$ pip freeze > requirements_test_py38.txt

(Test Py38) ~/virtualenvs$ deactivate

If you ever have to reinstall you can create a new virtual environment again and then run:

(Test Py38) ~/virtualenvs$ 
$ pip install -r requirements.txt

Installing New Packages

Installing, upgrading and managing packages is via the “pip” Python package manager. See pip help.   Use pip list and/or pip freeze to list your installed Python packages.

Use pip install <package_name> and pip uninstall <package_name> to install or remove Python packages. Example: Install the PyRNA package:

$ pip install PyRNA
Downloading PyRNA-1.0.0.dev4.tar.gz (5.2 kB)
Successfully built PyRNA
Successfully installed PyRNA-1.0.0.dev4

Use pip show to show infomation on installed Python packages:

$ pip show PyRNA
Name: PyRNA
Version: 1.0.0.dev4
Summary: View RNA secondary structure and BLAST search it for identification
Author: Jan Aspan
License: UNKNOWN

Legacy Python 2.7 Environment

Python 2 passed the end of its supported life on January 1st, 2020. Python 2.7 is no longer maintained. If you absolutely need a Python 2 environment then this is how to create it. There is only one version of Python 2 on the HPC, that is /usr/bin/python2.7, so that is what we will clone.

I store all my Python virtual environments in a subdirectory “virtualenvs” so I change to that directory first.

$ cd virtualenvs

Then use virtualenv to clone the python2.7 environment into the directory test_py27:

$ virtualenv --python=/usr/bin/python2.7  test_py27
  Running virtualenv with interpreter /usr/bin/python2.7
  Installing setuptools, pip, wheel...done.

Now activate and update it the same as a Python 3 environment.