Python Virtual Environments¶
This page covers installing your own version of Python 2 or Python 3 using
the virtualenv
command.
These are called “Python virtual environments” and are cloned from the
HPC’s system Python into your own home directory. They are about 250 MB in size.
This is much smaller than a Miniconda install.
See the page Install Miniconda Python.
Why should you use virtual Python environments?
Your own version of Python would be needed if your application or code requires
additional Python packages, or a specific version of Python or Python packages.
Importantly it also means that you control what versions of Python packages are
installed and they will only be upgraded if you do so. Our system Python upgrades
will not affect your Python virtual environment.
We will be using the command virtualenv. See virtualenv --help
What is the HPC’s System Python?¶
The HPC’s system Python is the one that would be invoked if you just typed
python2
or python3
.
hpcnode01 $ which python2
/usr/bin/python2
hpcnode01 $ which python3
/usr/bin/python3
hpcnode01 $ python2 --version
Python 2.7.17
hpcnode01 $ python3 --version
Python 3.6.8
It is these environments that will be cloned when you run virtualenv
and create your own Python environment.
Installing your own Python 3.x Environment¶
There are two Python 3 versions on the cluster: /usr/bin/python3.6
and
/usr/bin/python3.8
. You can use either one.
Let’s create a new Python 3 environment based on /usr/bin/python3.8
.
I store all my Python virtual environments in a subdirectory “virtualenvs” so I change to that directory first.
$ cd virtualenvs
Now use virtualenv
specifying the full path to the system Python executable that we
wish to clone and a directory where we wish install the cloned Python environment.
In the example below the python location is /usr/bin/python3.8
and the directory
test_py38
will be created.
Note: You will probably prefer to use a directory name for your Python
environment based on whatever project you are working on rather than test_py38
.
However do not use a name with spaces in it.
virtualenvs$ virtualenv --python=/usr/bin/python3.8 test_py38
(If you do not specify a path to a Python executable it will use the default
Python 3 which is /usr/bin/python3.6
.)
Now you need to “activate” this Python environment. Note: you must use
source test_py38/bin/activate
, you cannot run activate directly like
test_py38/bin/activate
.
virtualenvs$ source test_py38/bin/activate
(test_py38) ~/virtualenvs$
The shell prompt will now show the name of the Python environment currently active
i.e. (test_py38)
. The path to this new Python i.e. ~/test_py38/bin
will have been added
to the start of your existing PATH.
It will have also added a “VIRTUAL_ENV” environment variable.
(test_py38) $ which python
~/virtualenvs/test_py38/bin/python
(test_py38) $ python -V
Python 3.8.12
(test_py38) $ echo $VIRTUAL_ENV
/shared/homes/mlake/virtualenvs/test_py38
It’s probably a good idea at this stage to make sure that Python’s package manager “pip” is up-to-date:
(test_py38) $ python -m pip install --upgrade pip
To exit this Python environment use “deactivate”. This will return you to the default system Python environment.
(test_py38) ~$ deactivate
$
Using a Python Virtualenv with PBS¶
Within a PBS submission script you will need to activate your Python virtual environment, just the same as you would do at the command line. Add the line, like the one below, before the line which runs your python code.
source $HOME/virtualenvs/test_py38/bin/activate
and at the end of your PBS submission script, in the “Cleanup” section add:
deactivate
Those are the only changes required.
Upgrade pip and Other Packages¶
The first thing you should do when you create a Python virtual environment is to upgrade its Python package manager “pip”.
virtualenvs/$ source test_py38/bin/activate
(test_py38)$
(test_py38)$ python -m pip install --upgrade pip
You are using pip version 7.1.0, however version 19.1.1 is available.
Found existing installation: pip 7.1.0
Uninstalling pip-7.1.0:
Successfully uninstalled pip-7.1.0
Successfully installed pip-19.1.1
(test_py38)$
Now you should consider upgrading all the currently installed packages. You can get a list of Python packages which are outdated with:
(test_py38) $ pip list --outdated
Package Version Latest
---------- ------- ------
h5py 2.8.0 2.9.0
numpy 1.15.1 1.16.2
pip 18.0 19.0.3
scipy 1.1.0 1.2.1
setuptools 12.0.5 40.8.0
six 1.11.0 1.12.0
You can upgrade a package like “numpy” like this:
$ pip install --upgrade numpy
Another way to do this is to install the pip-review
package and use that:
$ pip install pip-review
$ pip-review --interactive
The interactive
option will ask you if you wish to upgrade for each package or all
the packages at once. You can also use the auto
option. See pip-review -h
.
Create a requirements.txt File¶
It’s really important to record what packages, and at what versions, are installed in
each virtual environment. Running pip freeze
will output a list of each installed
package and its version. This output can then be used to reinstall the packages again
into this virtual environment with pip install -r requirements.txt
.
Create your “requirements.txt” file:
virtualenvs$ source test_py38/bin/activate
(test_py38) ~/virtualenvs$
$ pip freeze > requirements_test_py38.txt
(test_py38) ~/virtualenvs$ deactivate
~/virtualenvs$
If you ever have to reinstall you can create a new virtual environment again and then run:
(test_py38) ~/virtualenvs$
$ pip install -r requirements.txt
Installing New Packages¶
Installing, upgrading and managing packages is via the “pip” Python package manager.
See pip help
. Use pip list
and/or pip freeze
to list your installed Python packages.
Use pip install <package_name>
and pip uninstall <package_name>
to install
or remove Python packages. Example: Install the PyRNA package:
$ pip install PyRNA
Downloading PyRNA-1.0.0.dev4.tar.gz (5.2 kB)
Successfully built PyRNA
Successfully installed PyRNA-1.0.0.dev4
Use pip show
to show infomation on installed Python packages:
$ pip show PyRNA
Name: PyRNA
Version: 1.0.0.dev4
Summary: View RNA secondary structure and BLAST search it for identification
Home-page: https://github.com/pypa/sampleproject
Author: Jan Aspan
Author-email: jan.aspan@gmail.com
License: UNKNOWN
Legacy Python 2.7 Environment¶
Python 2 passed the end of its supported life on January 1st, 2020.
Python 2.7 is no longer maintained. If you absolutely need a Python
2 environment then this is how to create it.
There is only one version of Python 2 on the HPC, that is /usr/bin/python2.7
,
so that is what we will clone.
I store all my Python virtual environments in a subdirectory “virtualenvs” so I change to that directory first.
$ cd virtualenvs
Then use virtualenv
to clone the python2.7
environment into the directory test_py27
:
$ virtualenv --python=/usr/bin/python2.7 test_py27
Running virtualenv with interpreter /usr/bin/python2.7
Installing setuptools, pip, wheel...done.
$
Now activate and update it the same as a Python 3 environment.