Job monitoring...

Jobs submitted to Gadi are given a jobID, this is shown to you as soon as it has been accepted and is a string of eight numbers, e.g. 12345678.

NCI encourages users to monitor their jobs at every stage, to monitor it's health and assist in detecting errors and failures. 

However, please refrain from checking your jobs excessively. Repeated queries will be considered attacks, especially in quick succession. Our recommendation is to query your jobs status a maximum of once every 10 minutes, this should be more than enough. 

• Queue Status

• CPU and Memory Utilisation

• Other Useful Monitoring Commands

• Files in Folder $PBS_JOBFS

To query job status, users run the command

$ qstat -swx <jobID> 

The command -swx is made up of:

-s: Summary format - shows queue level status rather than individual jobs

-w: Wide format - displays output in multiple columns

-x: Extended/Expanded format - includes additional details in the output

The screenshot below is in regards to job 12345678.gadi -pbs

1. User aaa777 submitted the job
2. To the normal-exec queue
3. They requested 48 cores and 190 GiB memory
4. It requested 2:00 hours and has been running for 0:35:21
5. The line at the bottom indicates when the job started, what Gadi node it is running on, 2697, and the space reserved on jobfs

You can go even further with this and run the command

$ qstat -fx <jobID> | less

In this case, -f means full/full information, by combining this with -x, we can print the complete job information. 

If you would like to see a list of qstat commands and their functions, please check the qstat manual by running 

$ man qstat

Users should continue to monitor their jobs, especially the utilisation rate. If users run into errors, this will be evident in a drop in utilisation rate. While a low utilisation rate is helpful for spotting the underuse of compute time, a 100% utilisation rate doesn't necessarily indicate the most efficient use of requested resources. Further enquiries can be made to check if performance can be improved. 

 To see how much CPU and memory your job has actually been using, run the command 

$ nqstat_anu <jobID>

This show us that the CPU ran at only 23% of the compute capacity of the 48 cores that were requested and that 36:47 has elapsed. 

It also shows the peak memory usage in the columns RSS and MEM.

Depending on the tasks running within this job, the percentage may increase as its lifespan continues. NCI recommends that users aim for at least 80% overall CPU utilisation rate.