Python Virtual Environments

This page covers installing your own version of Python 2 or Python 3 using the virtualenv command. These are called “Python virtual environments” and are cloned from the HPC’s system Python into your own home directory. They are about 250 MB in size. This is much smaller than a Miniconda install. See the page Install Miniconda Python.

First off, if you are writing Python code try to use a Python 3 version. This is because Python 2 reached the end of its life on January 1st, 2020. Python 2 versions are no longer maintained.

Why should you use virtual Python environments?
Your own version of Python would be needed if your application or code requires additional Python packages, or a specific version of Python or Python packages. Importantly it also means that you control what versions of Python packages are installed and they will only be upgraded if you do so. Our system Python upgrades will not affect your Python virtual environment.

We will be using the command virtualenv. See virtualenv --help

What is the HPC’s System Python?

The HPC’s system Python is the one that would be invoked if you just typed python2 or python3.

hpcnode01 $ which python2
/usr/bin/python2

hpcnode01 $ which python3
/usr/bin/python3

hpcnode01 $ python2 --version
Python 2.7.17

hpcnode01 $ python3 --version
Python 3.6.8

It is these environments that will be cloned when you run virtualenv and create your own Python environment.

Installing your own Python 2.7 Environment

Remember it’s preferable to be using Python 3. If you absolutely need a Python 2 environment then this is how to create it. There is only one version of Python 2 on the HPC, that is /usr/bin/python2.7, so that is what we will clone.

I store all my Python virtual environments in a subdirectory “virtualenvs” so I change to that directory first.

$ cd virtualenvs

Then use virtualenv to clone the python2.7 environment into the directory test_py27:

$ virtualenv --python=/usr/bin/python2.7  test_py27
  Running virtualenv with interpreter /usr/bin/python2.7
  Installing setuptools, pip, wheel...done.
$ 

Now activate and update it the same as a Python 3 environment.

Installing your own Python 3.x Environment

There are two Python 3 versions on the cluster: /usr/bin/python3.6 and /usr/bin/python3.8. You can use either one. Let’s create a new Python 3 environment based on /usr/bin/python3.6.

I store all my Python virtual environments in a subdirectory “virtualenvs” so I change to that directory first.

$ cd virtualenvs

Now use virtualenv specifying the full path to the system Python executable that we wish to clone and a directory where we wish install the cloned Python environment.

virtualenvs$ virtualenv --python=/usr/bin/python3.6 test_py36

(If you do not specify a path to a Python executable it will use the default Python 3 which is /usr/bin/python3.6.)

Now you need to “activate” this Python environment. Note: you must use source test_py36/bin/activate, you cannot run activate directly like test_py36/bin/activate.

 virtualenvs$ source test_py36/bin/activate 
(test_py36) ~/virtualenvs$ 

The shell prompt will now show the name of the Python environment currently active i.e. (test_py36). The path to this new Python i.e. ~/test_py36/bin will have been added to the start of your existing PATH. It will have also added a “VIRTUAL_ENV” environment variable.

(test_py36) $ which python
~/virtualenvs/test_py36/bin/python 

(test_py36) $ python -V
Python 3.6.8

(test_py36) $ echo $VIRTUAL_ENV
/shared/homes/mlake/virtualenvs/test_py36

To exit this python use “deactivate”. This will return you to the default system Python environment.

(test_py36) ~$ deactivate
$ 

Upgrade pip and Other Packages

The first thing you should do when you create a Python virtual environment is to upgrade its Python package manager “pip”.

virtualenvs/$ source test_py36/bin/activate
(test_py36)$ 

(test_py36)$ python -m pip install --upgrade pip
You are using pip version 7.1.0, however version 19.1.1 is available.
Found existing installation: pip 7.1.0
Uninstalling pip-7.1.0:
Successfully uninstalled pip-7.1.0
Successfully installed pip-19.1.1
(test_py36)$ 

Now you should consider upgrading all the currently installed packages. You can get a list of Python packages which are outdated with:

(test_py36) $ pip list --outdated
Package     Version  Latest 
----------  -------  ------ 
h5py        2.8.0    2.9.0  
numpy       1.15.1   1.16.2 
pip         18.0     19.0.3 
scipy       1.1.0    1.2.1  
setuptools  12.0.5   40.8.0 
six         1.11.0   1.12.0 

You can upgrade a package like “numpy” like this:

$ pip install --upgrade numpy

Another way to do this is to install the pip-review package and use that:

$ pip install pip-review
$ pip-review --interactive

The interactive option will ask you if you wish to upgrade for each package or all the packages at once. You can also use the auto option. See pip-review -h.

Installing New Packages

Installing, upgrading and managing packages is via the “pip” Python package manager. See pip help.   Use pip list and/or pip freeze to list your installed Python packages.

Use pip install <package_name> and pip uninstall <package_name> to install or remove Python packages. Example: Install the PyRNA package:

$ pip install PyRNA
Downloading PyRNA-1.0.0.dev4.tar.gz (5.2 kB)
Successfully built PyRNA
Successfully installed PyRNA-1.0.0.dev4

Use pip show to show infomation on installed Python packages:

$ pip show PyRNA
Name: PyRNA
Version: 1.0.0.dev4
Summary: View RNA secondary structure and BLAST search it for identification
Home-page: https://github.com/pypa/sampleproject
Author: Jan Aspan
Author-email: jan.aspan@gmail.com
License: UNKNOWN
This is custom footer