Python Virtual Environments¶
This page covers installing your own version of Python 3. These are called “Python virtual environments” and are cloned from the HPC’s system Python into your own home directory. They are about 250 MB in size. This is much smaller than a Miniconda install. See the page Install Miniconda Python.
Why should you use virtual Python environments?
Your own version of Python would be needed if your application or code requires
additional Python packages, or a specific version of Python or Python packages.
Importantly it also means that you control what versions of Python packages are
installed and they will only be upgraded if you do so. Our system Python upgrades
will not affect your Python virtual environment.
We will be using the command virtualenv. See virtualenv --help
What is the HPC’s System Python?¶
The HPC’s system Python is the one that would be invoked if you just typed
python2 or python3.
hpcnode01 $ which python2
/usr/bin/python2
hpcnode01 $ which python3
/usr/bin/python3
hpcnode01 $ python2 --version
Python 2.7.17
hpcnode01 $ python3 --version
Python 3.6.8
hpcnode01 $ python3.6 --version
Python 3.6.8
hpcnode01 $ python3.8 --version
Python 3.8.17
Any of these environments can be cloned to create your own Python environment based off the HPC’s Python system.
Installing your own Python 3.x Environment¶
There are two Python 3 versions on the cluster: /usr/bin/python3.6 and
/usr/bin/python3.8. You can use either one.
Let’s create a new Python 3.8 environment based on /usr/bin/python3.8.
I store all my Python virtual environments in a subdirectory “virtualenvs” so I change to that directory first.
$ cd virtualenvs
The command below is explained here:
we invoke the Python executable that we wish to clone i.e. /usr/bin/python3.8
we also need to load its virtual environment module i.e. -m venv
and we tell it where to create that cloned environment i.e. into a directory test_py38.
Optionally we can also tell it to use a more descriptive prompt when that
environment is loaded i.e. “Test Py3.8”.
$ /usr/bin/python3.8 -m venv test_py38 --prompt="Test Py3.8"
$ source test_py38/bin/activate
$ python -m pip install --upgrade pip
Note: for the directory where the environment will be created do not use any spaces i.e.
test_py38, however for the nicer prompt you can use spaces i.e. “Test Py3.8”.
You will probably prefer to use a directory name for your Python
environment based on whatever project you are working on rather than test_py38.
Now you need to “activate” this Python environment. Note: you must use
source test_py38/bin/activate, you cannot run activate directly like
test_py38/bin/activate.
virtualenvs$ source test_py38/bin/activate
(Test Py38) ~/virtualenvs$
The shell prompt will now show the name of the Python environment currently active
i.e. (Test Py38). The path to this new Python i.e. ~/test_py38/bin will have been added
to the start of your existing PATH.
It will have also added a “VIRTUAL_ENV” environment variable.
(Test Py38) $ which python
~/virtualenvs/test_py38/bin/python
(Test Py38) $ python -V
Python 3.8.12
(test_py38) $ echo $VIRTUAL_ENV
/shared/homes/mlake/virtualenvs/test_py38
It’s probably a good idea at this stage to make sure that Python’s package manager “pip” is up-to-date:
(Test Py38) $ python -m pip install --upgrade pip
To exit this Python environment use “deactivate”. This will return you to the default system Python environment.
(Test Py38) ~$ deactivate
$
Using a Python Virtualenv with PBS¶
Within a PBS submission script you will need to activate your Python virtual environment, just the same as you would do at the command line. Add the line, like the one below, before the line which runs your python code.
$ source $HOME/virtualenvs/test_py38/bin/activate
and at the end of your PBS submission script, in the “Cleanup” section add:
$ deactivate
Those are the only changes required.
Upgrade pip and Other Packages¶
The first thing you should do when you create a Python virtual environment is to upgrade its Python package manager “pip”.
virtualenvs/$ source test_py38/bin/activate
(test Py38)$
(Test Py38)$ python -m pip install --upgrade pip
You are using pip version 7.1.0, however version 19.1.1 is available.
Found existing installation: pip 7.1.0
Uninstalling pip-7.1.0:
Successfully uninstalled pip-7.1.0
Successfully installed pip-19.1.1
(Test Py38)$
Now you should consider upgrading all the currently installed packages. You can get a list of Python packages which are outdated with:
(Test Py38) $ pip list --outdated
Package Version Latest
---------- ------- ------
h5py 2.8.0 2.9.0
numpy 1.15.1 1.16.2
pip 18.0 19.0.3
scipy 1.1.0 1.2.1
setuptools 12.0.5 40.8.0
six 1.11.0 1.12.0
You can upgrade a package like “numpy” like this:
$ pip install --upgrade numpy
Another way to do this is to install the pip-review package and use that:
$ pip install pip-review
$ pip-review --interactive
The interactive option will ask you if you wish to upgrade for each package or all
the packages at once. You can also use the auto option. See pip-review -h.
Create a requirements.txt File¶
It’s really important to record what packages, and at what versions, are installed in
each virtual environment. Running pip freeze will output a list of each installed
package and its version. This output can then be used to reinstall the packages again
into this virtual environment with pip install -r requirements.txt.
Create your “requirements.txt” file:
virtualenvs$ source test_py38/bin/activate
(Test Py38) ~/virtualenvs$
$ pip freeze > requirements_test_py38.txt
(Test Py38) ~/virtualenvs$ deactivate
~/virtualenvs$
If you ever have to reinstall you can create a new virtual environment again and then run:
(Test Py38) ~/virtualenvs$
$ pip install -r requirements.txt
Installing New Packages¶
Installing, upgrading and managing packages is via the “pip” Python package manager.
See pip help. Use pip list and/or pip freeze to list your installed Python packages.
Use pip install <package_name> and pip uninstall <package_name> to install
or remove Python packages. Example: Install the PyRNA package:
$ pip install PyRNA
Downloading PyRNA-1.0.0.dev4.tar.gz (5.2 kB)
Successfully built PyRNA
Successfully installed PyRNA-1.0.0.dev4
Use pip show to show infomation on installed Python packages:
$ pip show PyRNA
Name: PyRNA
Version: 1.0.0.dev4
Summary: View RNA secondary structure and BLAST search it for identification
Home-page: https://github.com/pypa/sampleproject
Author: Jan Aspan
Author-email: jan.aspan@gmail.com
License: UNKNOWN
Legacy Python 2.7 Environment¶
Python 2 passed the end of its supported life on January 1st, 2020.
Python 2.7 is no longer maintained. If you absolutely need a Python
2 environment then this is how to create it.
There is only one version of Python 2 on the HPC, that is /usr/bin/python2.7,
so that is what we will clone.
I store all my Python virtual environments in a subdirectory “virtualenvs” so I change to that directory first.
$ cd virtualenvs
Then use virtualenv to clone the python2.7 environment into the directory test_py27:
$ virtualenv --python=/usr/bin/python2.7 test_py27
Running virtualenv with interpreter /usr/bin/python2.7
Installing setuptools, pip, wheel...done.
$
Now activate and update it the same as a Python 3 environment.