Accessing GPU Nodes

On our cluster we have some nodes with GPUs. The HPC Status page will indicate what nodes are GPU capable. The new node “hpccnode10” has two Nvidia V100 GPUs installled. The older node “ermdc23” has two Nvidia P100 GPUs installed.

Requesting GPU Nodes

In addition to the number of CPU cores required i.e. #PBS -l ncpus=1 just specify how many GPUs your job will require with #PBS -l ngpus=1. Usually just one GPU will be required. When you submit this job the PBS scheduler will allocate this job to a node that has a GPU.

Please don’t request a GPU unless you are actually runnning your code.

Logging into a GPU Node

Normally to run a job on the cluster you login to the login node and run qsub my_job_script.sh which contains PBS commands specifying the CPU, memory and time resources that you need. PBS then allocates your job the most appropriate node for your requirements. Generally you can’t login to nodes other than the login node. (See though Logging into Other Nodes.)

But in the case of GPUs to compile your CUDA program you might need to login to a GPU node as you need to have access to the CUDA libraries. You do this by submitting an interactive PBS job on the login node that logs you into a GPU node for a specified period of time. From there you can compile and test.

If your just compiling code you don’t need to request actual GPUs. You just need to be logged into a GPU node for your compiler to detect the CUDA libraries.

This command will log you into a GPU node for 30 minutes.

$ qsub -I -q gpuq -l walltime=00:30:00
hpccnode10 ~$

This command will log you into a specific GPU node for 30 minutes.

$ qsub -I -q gpuq -l host=hpccnode10 -l walltime=00:30:00
hpccnode10 ~$

Notice that the above did not request any GPUs. You do not need to use them if you are just compiling your code.

You will get 1 CPU core and 5GB of RAM as that is the default. The default walltime is 1 hour.

If you actually need to use a GPU for testing you can add that request to your interactive session request like this:

$ qsub -I -q gpuq -l select=1:ngpus=1 -l walltime=00:30:00
hpccnode10 ~$

In the above example we didn’t pick a specific node, we just asked for one GPU core and the PBS scheduler selected a node for us. If you need more CPU cores and memory for testing then then specify what you require like this:

$ qsub -I -q gpuq -l select=1:ngpus=1:ncpus=2:mem=15gb -l walltime=00:30:00
hpccnode10 ~$

When you are finished compiling or testing just type exit to logout of that shell.

Interactive logins will provide you with resources from the “Interactive Queue”. This is limited in wall time. See https://hpc.research.uts.edu.au/status/
For testing please use a minimal test data set and minimal cores, memory and time.

Interactive Login Using a Job Script

You can also place any of the PBS commands for an interactive job as described above into a script just like your normal job scripts. Create a script called gpu_access.sh. See the Example Interactive PBS Script at the end of this page. In this script set a walltime sufficient for you to do your task of compiling or testing. For instance if you set walltime=01:00:00 you will have 1 hour.

Now submit this script as a PBS interactive job (note the -I to invoke interactive mode).

$ qsub -I gpu_access.sh   

You will now have a shell on a GPU node. To exit the GPU node, just type “exit”. You will drop back to the login node. You will be also dropped back to the login node once your wall time is reached.

Here is an example gpu_access.sh script. Note that we have not requested any GPU cores i.e. no #PBS -l ngpus=1 because we just require an interactive node for compiling our program.

#!/bin/bash

# This is a simple PBS script that uses the "interactive" mode of PBS.
# See "Interactive-batch Jobs" in the "PBS User Guide".
#
# For interactive use you must submit this job with -I 
# qsub -I this_script.sh

#PBS -l ncpus=1
#PBS -l mem=5GB
#PBS -q gpuq

# Set your interactive wall time limit, hh:mm:ss
#PBS -l walltime=01:00:00

# Note: don't have any other commands below here! 
This is custom footer