Running Legacy Centos 6 Applications

This page describes how to use the Centos 6 Singularity image to run legacy Centos 6 applications under PBS.

Under /shared/eresearch/singularities/ you will find the Singularity image centos6_20200910.sif. Read the README.txtin that directory.

Running under PBS

Running your application with a PBS submission script and using singularity only involves a few changes. A typical template for your PBS submission script would be like this:

#!/bin/bash

#PBS -N test
#PBS -l ncpus=4
#PBS -l mem=5GB
#PBS -l walltime=00:10:00 

# Create a local scatch directory for your run
cd ${PBS_O_WORKDIR}
SCRATCH="/scratch/${USER}_${PBS_JOBID%.*}"
mkdir $SCRATCH

# Copy your input data to this scratch directory
cp YOUR_DATA $SCRATCH

# Start the Job
cd $SCRATCH
singularity exec /shared/eresearch/singularities/centos6_20200910.sif YOUR_PROGRAM

# Copy results and cleanup
mv $SCRATCH/* ${PBS_O_WORKDIR}
rmdir $SCRATCH

Replace YOUR_DATA and YOUR_PROGRAM with appropriate values.
In the script above instead of just running YOUR_PROGRAM we run it as singularity exec IMAGE_FILE_NAME YOUR_PROGRAM.

Example using samtools

Here is a quick example of using samtools. This is installed in the singularity image itself as its is quite a small application.

#!/bin/bash

#PBS -N samtools
#PBS -l ncpus=1
#PBS -l mem=5GB
#PBS -l walltime=00:5:00 

cd ${PBS_O_WORKDIR}
echo ""
echo "This is on a hpc node:"
/usr/bin/samtools --version

echo ""
echo "This is inside the Centos 6 singularity:"
singularity exec /shared/eresearch/singularities/centos6_20200910.sif samtools

Save this as run_sam.sh and you will get the two PBS files for stdout and stderr:

File: samtools.o42833

This is on a hpc node:
samtools 1.9
Using htslib 1.9
Copyright (C) 2018 Genome Research Ltd.

This is inside the Centos 6 singularity:

Notice it is missing the output of the samtools command in the singularity. This is just because samtools version 0.1.18 does not have a --version option and just using samtools sends its output to the stderr stream. So you will find the version information in the error file.

File: samtools.e42833

Program: samtools (Tools for alignments in the SAM format)
Version: 0.1.18 (r982:295)

Usage:   samtools <command> [options]

Command: view        SAM<->BAM conversion
         sort        sort alignment file
         mpileup     multi-way pileup
         depth       compute the depth
         faidx       index/extract FASTA
         tview       text alignment viewer
         index       index alignment
         idxstats    BAM index stats (r595 or later)
         fixmate     fix mate information
         flagstat    simple stats
         calmd       recalculate MD/NM tags and '=' bases
         merge       merge sorted alignments
         rmdup       remove PCR duplicates
         reheader    replace BAM header
         cat         concatenate BAMs
         targetcut   cut fosmid regions (for fosmid pool only)
         phase       phase heterozygotes

The particular run above was job number 42833.hpcnode0 and we can tell what node it ran on via:

hpcnode01 $ qstat -fx 42833.hpcnode0 | grep exec
exec_host = hpcnode03/1

So this ran on the newer hpcnodes running Centos 8.
Thus the very old samtools 0.1.18 can be run from inside the Centos 6 singularity image on the Centos 8 nodes.

Example using ddscat

Below is an example of running an application using singularity under PBS. Here we will use the application ddscat which can calculate the scattering and absorption of light by irregular particles and periodic arrangements of irregular particles.

Note: There are two versions of ddscat in /shared/opt/centos6/; ddscat-7.2.2 and a later ddscat-7.3.2. This latter one does work under Singularity so we will use ddscat-7.3.2.

First we note that this ddscat does not work under Centos 8, it is missing the libraries that it was compiled and linked against:

$ /shared/opt/centos6/ddscat-7.3.2/ddscat
/shared/opt/centos6/ddscat-7.3.2/ddscat: error while loading shared libraries: 
  libmpi_usempi.so.1: cannot open shared object file: No such file or directory
$

The library libmpi_usempi.so.1 is missing and the version of this library that is required is old and not available on Centos 8.

Lets run it under a singularity image that simulates a Centos 6 node. All of the files for this example are /shared/opt/centos6/ddscat_example/

This is the PBS submission script.

#!/bin/bash

# Example PBS submission script for singularity

#PBS -N test
#PBS -l ncpus=4
#PBS -l mem=5GB
#PBS -l walltime=00:10:00 

##PBS -m abe 
##PBS -M Your.Email@uts.edu.au

# Setup your data for the run
cd ${PBS_O_WORKDIR}
SCRATCH="/scratch/${USER}_${PBS_JOBID%.*}"
mkdir $SCRATCH
cp ddscat.par $SCRATCH
cp -r diel $SCRATCH

# Start the Job
cd $SCRATCH
singularity exec /shared/eresearch/singularities/centos6_20200910.sif \
    /shared/opt/centos6/ddscat-7.3.2/ddscat

# Copy results and cleanup
mv $SCRATCH/* ${PBS_O_WORKDIR}/output/
rmdir $SCRATCH

In your home directory or where you have example jobs directory, recursively copy the ddscat example to there:

$ cp -r /shared/opt/centos6/ddscat_example/ . 
$ cd ddscat_example/

Edit the run_job.sh script and change “Your.Email@uts.edu.au” to your own email. You don’t need to change anything else.

Submit the job:

$ qsub run_job.sh 
42772.hpcnode0

This run will take a bit less than 1 minute and use 4 cores and less than 5GB of RAM

You will find the output data in the output directory and you will have the two PBS files for the error and output streams. Note that for ddscat the program incorrectly directs the normal output to the error stream! Don’t worry about that. The important thing is that ddscat runs successfully using singularity.

Using Modules

If you need to load any modules within singularity then first do this:

Singularity> source /etc/profile.d/modules.sh
This is custom footer