Using Miniconda Python¶
Miniconda Python is a Python distribution that can be installed in your home directory. See https://conda.io/en/latest/
Warning
There is also an Anaconda Python distribution (https://www.anaconda.com/products/distribution).
Do not use this!
This includes a large number of data-science packages which you will not use.
It’s very very large in size.
Also consider using a Python virtualenv as described here: Python Virtual Environments. These are about 250 MB in size. A Miniconda Python 3 is about 10 GBytes in size.
Download Miniconda and Check the SHA256 Sum¶
Download the “Miniconda3 Linux 64-bit” from its download page: https://conda.io/en/latest/miniconda.html. The filename will be “Miniconda3-latest-Linux-x86_64.sh”.
If you are on your local machine you can just “click” that file to download it to your local machine. If you are on the HPC login node download it with “wget”.
$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
On the download page is the “SHA256 hash” for the installers”. It’s important to check that you have an official and verified Miniconda install script before you run it. Search for the filename that you downloaded and find its SHA265 sum. On the cluster check the SHA256 sum like this:
$ sha256sum Miniconda3-latest-Linux-x86_64.sh
00938c3534750......f4a9269db6 Miniconda3-latest-Linux-x86_64.sh
It should match. If it does not match do not run the installer! Contact us.
Install Miniconda¶
Warning
During the install of Miniconda it will ask you if you wish to “initialize Miniconda3”.
Read this section Miniconda’s “initialize” Changes
to understand the changes.
If you answer “yes” then changes will be made to the end of your .bashrc file.
This can cause problems.
I would strongly advise you to answer “no”.
To mitigate problems with users still doing the above I have set the following:
Note
A system wide environment variable, CONDA_AUTO_ACTIVATE_BASE=false has been set.
This will mean that your base Conda environment will not automatically load when you login.
You can load your base Conda environment with conda activate base.
This will be installed into your own home directory:
$ bash Miniconda3-latest-Linux-x86_64.sh
Welcome to Miniconda3
In order to continue the installation process, please review the license agreement.
Please, press ENTER to continue
.....
Miniconda3 will now be installed into this location:
/shared/homes/xxxxxxx/miniconda3
.....
PREFIX=/shared/homes/xxxxxx/miniconda3
installing: base environment ...
.......
The installer will then ask you:
Do you wish to update your shell profile to automatically initialize conda?
This will activate conda on startup and change the command prompt when activated.
...
[no] >>>
See the warning at the top of this page!
Miniconda is now installed. We just need to activate it.
Activating & Deactivating your Base Conda Environment¶
During the installation, if you you followed the advice above, you would not
have allowed the installer to “initialize Miniconda3”,
In which case you need to run the command below so the conda commands such as
conda activate, conda info, conda create etc can be found. It won’t
change the default Python, which is still the system /usr/bin/python3.
If you allowed the Miniconda installer to “initialize Miniconda3” then you
won’t need to run the command below.
$ source $HOME/miniconda3/etc/profile.d/conda.sh
$
Now you can activate either the base conda environment as below or any other conda environment you have installed.
$ conda activate base
(base) $
Notice our prompt has changed from $ to (base)$, this means we are in the
base conda python environment.
To deactivate your base conda environemnt type:
(base) $ conda deactivate
$
You should keep your base conda environment “minimal” i.e. don’t install lots of software into your base conda environment. Create separate, project based, conda environments for your software.
See the section Creating Conda Environments.
Activating & Deactivating your Project Conda Environments¶
You can activate and deactivate your own conda environments at any time like this. I have included the command for showing which python is currently active to make it clear what’s happening.
$ which python3
/usr/bin/python3 <-- This is the system Python.
$
If we had a conda environment called “project_A” we would
activate this like so:
$ source /shared/homes/XXXXXX/miniconda3/etc/profile.d/conda.sh
$ conda activate project_A
(project_A) $
(project_A) $ which python3
/shared/homes/XXXXXX/miniconda3/bin/python3 <-- This is your Python.
(project_A) $
(project_A) $ conda deactivate
$
$ which python3
/usr/bin/python3 <-- Back to the system Python.
$
Within a PBS submission script you can activate your project based conda environment like this:
#!/bin/bash
#PBS -N primes
#PBS -l ncpus=4
#PBS -l mem=20gb
#PBS -l walltime=00:40:00
# Change to the PBS working directory where qsub was started from.
cd ${PBS_O_WORKDIR}
# Activate python for project A
source /shared/homes/XXXXXX/miniconda3/etc/profile.d/conda.sh
conda activate project_A
# Run my python program.
path_to/my_program
Creating Conda Environments¶
This is one of the most useful features of Miniconda – the ability to create several completely independent conda environments. Rather than using the “base” Miniconda python we can create a new Python environment specific for what we wish to do. For instance we might have two projects; “Project A” which requires Python 3.6 and a set of Python modules and “Project B” that requires the newer Python 3.8 with a different set of python modules.
Let’s create a conda environment for “Project A” which will use Python 3.6. Note we will not use spaces in the directory name, spaces in file and directory names cause problems.
(base) $ conda create --name project_A python=3.6
.....
environment location: /shared/homes/XXXXXX/miniconda3/envs/project_A
#
# To activate this environment, use
#
# $ conda activate project_A
#
# To deactivate an active environment, use
#
# $ conda deactivate
(base) $ conda activate project_A
(project_A) $
You can run conda list to see what modules are already installed.
Run conda install numpy to install the Numpy package etc.
(project_A) $ conda deactivate
(base) $
Now let’s create Project B which will be a Python 3.8 environment.
(base) $ conda create --name project_B python=3.8
(project_B)
(project_B) conda install imageio
....
(project_A) $ conda deactivate
(base) $
Running conda list will show you that we now have different Python packages installed
in project_A and project_B.
(base)$ conda list -n project_A
(base)$ conda list -n project_B
You can remove a Miniconda environment with:
$ conda env remove -n project_B
Updating your Miniconda Environments¶
You should regularly update your base conda environment and any project environments. Miniconda does not use the standard “pip” Python packaging manager, it uses it’s own package manager called “conda”.
To make sure your base environment is at the latest version do a conda update conda from
within the base distribution.
(base) $ conda update conda
Collecting package metadata (current_repodata.json): done
Solving environment: done
...
The following packages will be UPDATED:
cryptography 2.6.1-py37h1ba5d50_0 --> 2.7-py37h1ba5d50_0
pip 19.0.3-py37_0 --> 19.1.1-py37_0
sqlite 3.27.2-h7b6447c_0 --> 3.28.0-h7b6447c_0
urllib3 1.24.1-py37_0 --> 1.24.2-py37_0
Proceed ([y]/n)? y
....
To update your “Project A” environment it’s similar to updating the base environment. First activate “Project A”. Then save the current list of modules (in case we need to revert) and then update them all.
(project_A)$ conda list --export > conda_export_2019.01.10.txt
(project_A)$ conda update --all
You should do this regularly within each of your conda environments.
Keep your Miniconda Environments Clean¶
During operation Conda will generate many temporary files. These files are cached and can build up over time. Many of them will be unnecessary and can be removed. This can save a lot of disk space. Make sure you read the documentation for the clean command here: https://docs.conda.io/projects/conda/en/latest/commands/clean.html
Start our Miniconda3:
$ source miniconda3/bin/activate
(base)$
As well as the HTML documentation above you can also read the local conda documentation:
(base)$ conda clean -h
First check how much disk space your Miniconda3 currently uses.
Read the “man” page for the du command for what these command line options do (man du).
(base)$ du -skh miniconda3/
56G miniconda3/
Here my Miniconda3 is using 56 Gbytes of disk space. Run the conda clean command.
This will remove the index cache, lock files, unused cache packages, tarballs, and logfiles.
(base)$ conda clean --all
Will remove 627 (37.11 GB) package(s).
......
Now see how much disk space Miniconda uses:
(base)$ du -skh miniconda3/
19G miniconda3/
The conda clean -a has saved me 56GB - 19GB = 37 GB !!
Now you can deactivate your Miniconda3.
(base) hpcnode01 ~/$ conda deactivate
hpcnode01 ~/$
Thanks to one of our HPC users, Sid Krishnan, for this great tip on managing Miniconda installations.
Useful Miniconda Commands¶
Obtain help on using conda with conda -h or `conda
$ conda config ⟸ Creates a ~/.condarc file the first time it is run.
$ conda config --show-sources ⟸ Shows the channels that provide the packages.
$ conda info ⟸ Display information about current conda install.
$ conda env list ⟸ Lists your conda environments.
$ conda list ⟸ Lists installed Python modules.
$ conda list --export ⟸ Save package list for future use.
$ conda clean -all ⟸ Remove cache files.
Miniconda’s “initialize” Changes¶
During the install you are asked:
Do you wish the installer to initialize Miniconda3
by running conda init? [yes|no]
[no] >>>
If you answer “yes” then installer will add extra commands to the end of your
.bashrc file. In this case the “conda” commands will be available when you login.
To remove those changes see the section below: How do I Revert Changes to my .bashrc?.
If you answer “no” then the installer will then display:
You have chosen to not have conda modify your shell scripts at all.
To activate conda's base environment in your current shell session:
eval "$(/shared/homes/XXXXXX/miniconda3/bin/conda shell.YOUR_SHELL_NAME hook)"
To install conda's shell functions for easier access, first activate, then:
conda init
If you'd prefer that conda's base environment not be activated on startup,
set the auto_activate_base parameter to false:
conda config --set auto_activate_base false
Thank you for installing Miniconda3!
$
The above suggests to activate your conda environment using “YOUR_SHELL_NAME”
and as your shell is probably “bash” you would do this:
eval "$(/shared/homes/XXXXXX/miniconda3/bin/conda shell.bash hook)"
But it’s easier and equivalent to just do this:
$ source $HOME/miniconda3/etc/profile.d/conda.sh
$
The above will just change your environment so the conda commands, such as
conda activate, conda info, conda create, conda list can be found.
It won’t change the default Python, which is still the system /usr/bin/python3.
How do I Revert Changes to my .bashrc?¶
If you have let the Miniconda installer run conda init it will have modified
your .bashrc file. To remove those changes just open up your .bashrc file
with a text editor and comment out the all lines between the “conda initialize”
labels like this:
## >>> conda initialize >>>
# .....
# .....
## <<< conda initialize <<<
Save and exit, then logout and log back in again.