Running Array Jobs

In this section we will look at running PBS array jobs. Array jobs are useful if you have a large number of jobs that you need to run with different, but calculatable, parameters.

Copy the Example Scripts

There is an example script that you can use to practice submitting a short array job in /shared/eresearch/pbs_job_examples/job_arrays/. Copy this into your own directory using the following commands:

$ cd            <-- This will take you to the top of your home directory. 
$ mkdir jobs    <-- This creates the directory "jobs".
$ cd jobs       <-- This changes into the directory "jobs".
$ cp -r /shared/eresearch/pbs_job_examples/job_arrays .   <-- Don't forget the dot at the end.
$ cd job_arrays     <-- This changes into the new "job_arrays" directory.

This will have recursively copied the directory job_arrays and its contents to your own directory. You will now be in that directory and you can have a look at the scripts there.

Submitting an Array Job

An an array job is submitted by using either a -J start-end:step specification to qsub or by including a #PBS -J start-end:step within your PBS submission script. In this example we will be specifying the array job specs on the qsub command line.

Submitting the script to PBS will return your PBS_JOBID for the array job, for example:

$ qsub job_array_script -J 1-10:2
28846[].hpcnode0

The “array job” will consist of “sub-jobs” and each of those will have the PBS_JOBID and a unique PBS_ARRAY_INDEX value within the brackets, for example:

28846[1].hpcnode0
28846[2].hpcnode0
28846[3].hpcnode0
28846[4].hpcnode0
28846[5].hpcnode0

When your job starts this PBS_ARRAY_INDEX value will be available within your job submission script. It’s up to you how you use it. For instance, you can use it to specify parameters to scripts or as a parameter to specify the names of your input files or name your output files.

The PBS directives that specify the resources will apply to EACH individual job not all the jobs together.

Checking your Array Jobs Status

The status of the sub-jobs is not displayed by default. For example, the following qstat options shows the job array as a single job: qstat -a or qstat -J.

$ qstat -a
                                                            Req'd  Req'd   Elap
Job ID     Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
---------- -------- -------- ---------- ------ --- --- ------ ----- - -----
230008[]   999777   defaultq test          --    1   3   16gb 00:12 Q   -- 

When the status (“S” column) shows “Q” then, like non-array jobs, the job is queued.
If the status (“S” column) shows “B” then this indicates that at least one sub-job has left the “Q” (queued) state and is running or has run, but not all sub-jobs have run.

To check the status of the sub-jobs, use either the -Jt option or the -t option with an array specified, for example:

$ qstat -Jt
                                                            Req'd  Req'd   Elap
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
230008[10200].h 999777   defaultq test          --    1   3   16gb 00:12 Q   -- 
230008[10201].h 999777   defaultq test          --    1   3   16gb 00:12 Q   -- 
230008[10202].h 999777   defaultq test          --    1   3   16gb 00:12 Q   -- 

or

$ qstat -t 230008[].hpcnode0
                                                            Req'd  Req'd   Elap
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
230008[].hpcnod 119966   defaultq cooc.         --    1   3   16gb 00:12 Q   -- 
230008[10200].h 119966   defaultq cooc.         --    1   3   16gb 00:12 Q   -- 
230008[10201].h 119966   defaultq cooc.         --    1   3   16gb 00:12 Q   -- 

Deleting an Array Job

To delete an array job use the qdel command and specify the array job ID or the sub-job ID i.e.:

$ qdel 28846[].hpcnode0 

or

$ qdel 28846[5].hpcnode0
This is custom footer