Checking GPU Usage¶
You can obtain a basic information on the NVIDA GPU and its current usage
using NVIDIA’s “System Management Interface” program
nvidia-smi. Look at its man page for details man nvidia-smi,
or run it with the -h option i.e. nvidia-smi -h for help.
See Checking Device Capability to find out detailed information on the card.
Another alternative to Nvidia’s monitoring tool is the interactive Nvidia GPU process viewer, see Using nvitop
Using nvidia-smi¶
Note
Please: Do not use the nvidia-smi command with the -l option, nor
with the watch command. Continuously running nvidia-smi using a loop consumes
GPU resources and will slow down everyone elses jobs.
You need to be logged into a GPU node to run this command. Here is an example of its output.
GPUNode $
$ nvidia-smi
Mon Jun 28 14:13:56 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.19.01 Driver Version: 465.19.01 CUDA Version: 11.3 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla V100-PCIE... Off | 00000000:3B:00.0 Off | 0 |
| N/A 61C P0 155W / 250W | 17289MiB / 32510MiB | 91% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-PCIE... Off | 00000000:D8:00.0 Off | 0 |
| N/A 62C P0 152W / 250W | 17289MiB / 32510MiB | 89% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 57573 C python 17283MiB |
| 1 N/A N/A 57573 C python 17283MiB |
+-----------------------------------------------------------------------------+
$
You can see that this node has two Tesla V100 GPUs installed. Both are running at about 50% memory usage and 90% GPU utilisation.
You can get a list of the two GPUs and their UUIDs with this:
hpcnode10 $ nvidia-smi --list-gpus
GPU 0: Tesla V100-PCIE-32GB (UUID: GPU-37f061b1-7948-e188-56a7-d30f5e0ffc70)
GPU 1: Tesla V100-PCIE-32GB (UUID: GPU-151b0546-4c5b-039a-e1e2-0acaa0098909)
You can specify what information you woud like to see by using the --query option
with --display parameters e.g.:
$ nvidia-smi -q -d MEMORY,COMPUTE,UTILIZATION
The above will show the data for both GPUs. If you only wish to see the information for a specific GPU then you can specify the UUID to query:
$ nvidia-smi -q -d MEMORY,COMPUTE,UTILIZATION -i GPU-37f061b1-7948-e188-56a7-d30f5e0ffc70 --loop=600
Notice in the above example I have also used the --loop option. This can be
very useful but please not use this continuously with small time intervals.
For all the details on this command see the manual pages man nvidia-smi.
Using nvitop¶
Another alternative to Nvidia’s monitoring tool is the interactive Nvidia GPU process viewer from: https://github.com/XuehaiPan/nvitop. The requirements for this are already installed on the HPC GPU nodes. Just go to the “Installation” section or read how I installed it below.
It needs to be installed into your own local Python environment.
If your using Miniconda you will need to load your Base Miniconda Python environment
and then load your Project Conda Environment.
(For a refresher on python environments see here:
https://hpc.research.uts.edu.au/software_general/python/python_miniconda/)
Be on a GPU node and load CUDA:
GPU node ~/$ module load cuda-latest
Load the base Miniconda and load your project environment. In this case mine is called “pytorch”:
$ source $HOME/miniconda3/etc/profile.d/conda.sh
$ conda activate pytorch
(pytorch)$
Now install nvitop:
$ pip3 install nvitop
Now you can just type nvitop to run it. Type nvitop -h for some short usage help.
The full documentation for it is on Xuehai Pan’s website as above.
My version of nvitop is 1.3.2