Skip to content

Python Virtual Environments

This page covers installing your own version of Python 2 or Python 3 using the virtualenv command. These are called “Python virtual environments” and are cloned from the HPC’s system Python into your own home directory. They are about 250 MB in size. This is much smaller than a Miniconda install. See the page Install Miniconda Python.

Why should you use virtual Python environments?
Your own version of Python would be needed if your application or code requires additional Python packages, or a specific version of Python or Python packages. Importantly it also means that you control what versions of Python packages are installed and they will only be upgraded if you do so. Our system Python upgrades will not affect your Python virtual environment.

We will be using the command virtualenv. See virtualenv --help

What is the HPC’s System Python?

The HPC’s system Python is the one that would be invoked if you just typed python2 or python3.

hpcnode01 $ which python2
/usr/bin/python2

hpcnode01 $ which python3
/usr/bin/python3

hpcnode01 $ python2 --version
Python 2.7.17

hpcnode01 $ python3 --version
Python 3.6.8

It is these environments that will be cloned when you run virtualenv and create your own Python environment.

Installing your own Python 3.x Environment

There are two Python 3 versions on the cluster: /usr/bin/python3.6 and /usr/bin/python3.8. You can use either one. Let’s create a new Python 3 environment based on /usr/bin/python3.8.

I store all my Python virtual environments in a subdirectory “virtualenvs” so I change to that directory first.

$ cd virtualenvs

Now use virtualenv specifying the full path to the system Python executable that we wish to clone and a directory where we wish install the cloned Python environment. In the example below the python location is /usr/bin/python3.8 and the directory test_py38 will be created.

Note: You will probably prefer to use a directory name for your Python environment based on whatever project you are working on rather than test_py38. However do not use a name with spaces in it.

virtualenvs$ virtualenv  --python=/usr/bin/python3.8  test_py38

(If you do not specify a path to a Python executable it will use the default Python 3 which is /usr/bin/python3.6.)

Now you need to “activate” this Python environment. Note: you must use source test_py38/bin/activate, you cannot run activate directly like test_py38/bin/activate.

virtualenvs$ source test_py38/bin/activate 
(test_py38) ~/virtualenvs$ 

The shell prompt will now show the name of the Python environment currently active i.e. (test_py38). The path to this new Python i.e. ~/test_py38/bin will have been added to the start of your existing PATH. It will have also added a “VIRTUAL_ENV” environment variable.

(test_py38) $ which python
~/virtualenvs/test_py38/bin/python 

(test_py38) $ python -V
Python 3.8.12

(test_py38) $ echo $VIRTUAL_ENV
/shared/homes/mlake/virtualenvs/test_py38

It’s probably a good idea at this stage to make sure that Python’s package manager “pip” is up-to-date:

(test_py38) $ python -m pip install --upgrade pip

To exit this Python environment use “deactivate”. This will return you to the default system Python environment.

(test_py38) ~$ deactivate
$

Using a Python Virtualenv with PBS

Within a PBS submission script you will need to activate your Python virtual environment, just the same as you would do at the command line. Add the line, like the one below, before the line which runs your python code.

source $HOME/virtualenvs/test_py38/bin/activate

and at the end of your PBS submission script, in the “Cleanup” section add:

deactivate

Those are the only changes required.

Upgrade pip and Other Packages

The first thing you should do when you create a Python virtual environment is to upgrade its Python package manager “pip”.

virtualenvs/$ source test_py38/bin/activate
(test_py38)$ 

(test_py38)$ python -m pip install --upgrade pip
You are using pip version 7.1.0, however version 19.1.1 is available.
Found existing installation: pip 7.1.0
Uninstalling pip-7.1.0:
Successfully uninstalled pip-7.1.0
Successfully installed pip-19.1.1
(test_py38)$ 

Now you should consider upgrading all the currently installed packages. You can get a list of Python packages which are outdated with:

(test_py38) $ pip list --outdated
Package     Version  Latest 
----------  -------  ------ 
h5py        2.8.0    2.9.0  
numpy       1.15.1   1.16.2 
pip         18.0     19.0.3 
scipy       1.1.0    1.2.1  
setuptools  12.0.5   40.8.0 
six         1.11.0   1.12.0 

You can upgrade a package like “numpy” like this:

$ pip install --upgrade numpy

Another way to do this is to install the pip-review package and use that:

$ pip install pip-review
$ pip-review --interactive

The interactive option will ask you if you wish to upgrade for each package or all the packages at once. You can also use the auto option. See pip-review -h.

Create a requirements.txt File

It’s really important to record what packages, and at what versions, are installed in each virtual environment. Running pip freeze will output a list of each installed package and its version. This output can then be used to reinstall the packages again into this virtual environment with pip install -r requirements.txt.

Create your “requirements.txt” file:

virtualenvs$ source test_py38/bin/activate 
(test_py38) ~/virtualenvs$

$ pip freeze > requirements_test_py38.txt

(test_py38) ~/virtualenvs$ deactivate
~/virtualenvs$

If you ever have to reinstall you can create a new virtual environment again and then run:

(test_py38) ~/virtualenvs$ 
$ pip install -r requirements.txt

Installing New Packages

Installing, upgrading and managing packages is via the “pip” Python package manager. See pip help.   Use pip list and/or pip freeze to list your installed Python packages.

Use pip install <package_name> and pip uninstall <package_name> to install or remove Python packages. Example: Install the PyRNA package:

$ pip install PyRNA
Downloading PyRNA-1.0.0.dev4.tar.gz (5.2 kB)
Successfully built PyRNA
Successfully installed PyRNA-1.0.0.dev4

Use pip show to show infomation on installed Python packages:

$ pip show PyRNA
Name: PyRNA
Version: 1.0.0.dev4
Summary: View RNA secondary structure and BLAST search it for identification
Home-page: https://github.com/pypa/sampleproject
Author: Jan Aspan
Author-email: jan.aspan@gmail.com
License: UNKNOWN

Legacy Python 2.7 Environment

Python 2 passed the end of its supported life on January 1st, 2020. Python 2.7 is no longer maintained. If you absolutely need a Python 2 environment then this is how to create it. There is only one version of Python 2 on the HPC, that is /usr/bin/python2.7, so that is what we will clone.

I store all my Python virtual environments in a subdirectory “virtualenvs” so I change to that directory first.

$ cd virtualenvs

Then use virtualenv to clone the python2.7 environment into the directory test_py27:

$ virtualenv --python=/usr/bin/python2.7  test_py27
  Running virtualenv with interpreter /usr/bin/python2.7
  Installing setuptools, pip, wheel...done.
$ 

Now activate and update it the same as a Python 3 environment.