This page describes how to use the Centos 6 Singularity image to run legacy Centos 6 applications under PBS.
Under /shared/opt/centos6/
you will find the Singularity image
centos6_20200910.sif
. Read the README.txt
in that directory.
Running your application with a PBS submission script and using singularity only involves a few changes. A typical template for your PBS submission script would be like this:
#!/bin/bash
#PBS -N test
#PBS -l ncpus=4
#PBS -l mem=5GB
#PBS -l walltime=00:10:00
# Create a local scatch directory for your run
cd ${PBS_O_WORKDIR}
SCRATCH="/scratch/${USER}_${PBS_JOBID%.*}"
mkdir $SCRATCH
# Copy your input data to this scratch directory
cp YOUR_DATA $SCRATCH
# Start the Job
cd $SCRATCH
singularity exec /shared/opt/centos6/centos6_20200910.sif YOUR_PROGRAM
# Copy results and cleanup
mv $SCRATCH/* ${PBS_O_WORKDIR}
rmdir $SCRATCH
Replace YOUR_DATA and YOUR_PROGRAM with appropriate values.
In the script above instead of just running YOUR_PROGRAM
we run it as
singularity exec IMAGE_FILE_NAME YOUR_PROGRAM
.
Here is a quick example of using samtools. This is installed in the singularity image itself as its is quite a small application.
#!/bin/bash
#PBS -N samtools
#PBS -l ncpus=1
#PBS -l mem=5GB
#PBS -l walltime=00:5:00
cd ${PBS_O_WORKDIR}
echo ""
echo "This is on a hpc node:"
/usr/bin/samtools --version
echo ""
echo "This is inside the Centos 6 singularity:"
singularity exec /shared/opt/centos6/centos6_20200910.sif samtools
Save this as run_sam.sh
and you will get the two PBS files for stdout and stderr:
File: samtools.o42833
This is on a hpc node:
samtools 1.9
Using htslib 1.9
Copyright (C) 2018 Genome Research Ltd.
This is inside the Centos 6 singularity:
Notice it is missing the output of the samtools command in the singularity.
This is just because samtools version 0.1.18 does not have a --version
option and
just using samtools
sends its output to the stderr stream. So you will find the
version information in the error file.
File: samtools.e42833
Program: samtools (Tools for alignments in the SAM format)
Version: 0.1.18 (r982:295)
Usage: samtools <command> [options]
Command: view SAM<->BAM conversion
sort sort alignment file
mpileup multi-way pileup
depth compute the depth
faidx index/extract FASTA
tview text alignment viewer
index index alignment
idxstats BAM index stats (r595 or later)
fixmate fix mate information
flagstat simple stats
calmd recalculate MD/NM tags and '=' bases
merge merge sorted alignments
rmdup remove PCR duplicates
reheader replace BAM header
cat concatenate BAMs
targetcut cut fosmid regions (for fosmid pool only)
phase phase heterozygotes
The particular run above was job number 42833.hpcnode0 and we can tell what node it ran on via:
hpcnode01 $ qstat -fx 42833.hpcnode0 | grep exec
exec_host = hpcnode03/1
So this ran on the newer hpcnodes running Centos 8.
Thus the very old samtools 0.1.18 can be run from inside the Centos 6
singularity image on the Centos 8 nodes.
Below is an example of running an application using singularity under PBS. Here
we will use the application ddscat
which can calculate the scattering and
absorption of light by irregular particles and periodic arrangements of irregular particles.
Note: There are two versions of ddscat in /shared/opt/centos6/
; ddscat-7.2.2
and
a later ddscat-7.3.2
. This latter one does work under Singularity so we will use
ddscat-7.3.2
.
First we note that this ddscat does not work under Centos 8, it is missing the libraries that it was compiled and linked against:
$ /shared/opt/centos6/ddscat-7.3.2/ddscat
/shared/opt/centos6/ddscat-7.3.2/ddscat: error while loading shared libraries:
libmpi_usempi.so.1: cannot open shared object file: No such file or directory
$
The library libmpi_usempi.so.1
is missing and the version of this library
that is required is old and not available on Centos 8.
Lets run it under a singularity image that simulates a Centos 6 node.
All of the files for this example are /shared/opt/centos6/ddscat_example/
This is the PBS submission script.
#!/bin/bash
# Example PBS submission script for singularity
#PBS -N test
#PBS -l ncpus=4
#PBS -l mem=5GB
#PBS -l walltime=00:10:00
##PBS -m abe
##PBS -M Your.Email@uts.edu.au
# Setup your data for the run
cd ${PBS_O_WORKDIR}
SCRATCH="/scratch/${USER}_${PBS_JOBID%.*}"
mkdir $SCRATCH
cp ddscat.par $SCRATCH
cp -r diel $SCRATCH
# Start the Job
cd $SCRATCH
singularity exec /shared/opt/centos6/centos6_20200910.sif \
/shared/opt/centos6/ddscat-7.3.2/ddscat
# Copy results and cleanup
mv $SCRATCH/* ${PBS_O_WORKDIR}/output/
rmdir $SCRATCH
In your home directory or where you have example jobs directory, recursively copy the ddscat example to there:
$ cp -r /shared/opt/centos6/ddscat_example/ .
$ cd ddscat_example/
Edit the run_job.sh
script and change “Your.Email@uts.edu.au” to your own email.
You don’t need to change anything else.
Submit the job:
$ qsub run_job.sh
42772.hpcnode0
This run will take a bit less than 1 minute and use 4 cores and less than 5GB of RAM
You will find the output data in the output directory and you will have the two PBS files for the error and output streams. Note that for ddscat the program incorrectly directs the normal output to the error stream! Don’t worry about that. The important thing is that ddscat runs successfully using singularity.
If you need to load any modules within singularity then first do this:
Singularity> source /etc/profile.d/modules.sh