FAQ for PBS jobs
Listed below are some of the common problems that you may encounter when running a PBS job.
Please check this page to see if your issue is listed here and has a solution.
On this page
What does 'exceeded memory allocation' mean?
If your job exceeds the memory request you gave in your PBS job script, you may receive a message like the following:
------------------------------------------------------------------------ Job 1234.gadi-pbs has exceeded memory allocation on node gadi-cpu-clx-0001 Process "bash", pid 21733, rss 1744896, vmem 9965568 Process "memhog", pid 21763, rss 524279808, vmem 10743767040 ------------------------------------------------------------------------
This message indicates that your job is using more memory than you requested in your job script's #PBS -lmem directive, or the amount you requested in your qsub's -lmem option.
When your job exceeds the memory request given on any one of the nodes in the job, the message above will be produced, and it is likely that the kernel will kill one of the processes in your job. Which process is killed depends on the memory pressure, which process is attempting to allocate more memory, and potentially other factors. This will likely cause your job to fail.
How is memory allocated to my job?
When you submit a PBS job you let PBS know the maximum amount of memory your job will use across all nodes for the job.
The total memory request for your job is divided by the number of nodes your job is requesting and allocated evenly to each node.
For example, if you request 384 CPUs in the normal queue, and 128GB of RAM, your job will be using 8 nodes (there are 48 CPU cores in each node in the normal queue on Gadi) and 16GB of RAM will be allocated on each of the 8 nodes.
How do I read the memory request exceeded message?
The first line of the message tells you which job exceeded the memory allocation, and what node the memory allocation was exceeded on. This may be helpful in determining which of the nodes in a multi-noded job is exceeding memory if there is a memory imbalance between nodes.
The subsequent lines indicates every process running as part of your job on that node. The process name 'bash' and process ID, 'pid 21733' are indicated and help identify which process in your job is using significant memory, or is not balanced with other processes.
The RSS value is the "resident set size" or actively used memory for that process in bytes. The vmem value is the virtual memory address space of the process; this may be significantly larger than your memory usage as virtual memory address space may be allocated by some processes but never actually used.
Why does the job summary at the end of my output not show the same usage as the exceeded message?
The memory usage in the job summary at the end of your job output is generated by sampling the job's overall memory usage on a regular basis.
If your memory usage has spiked upwards since the last sample, and your job is killed by the kernel for exceeding your memory allocation, the memory usage shown in the job summary will be less than your final memory usage.
The memory usage shown in the exceeded 'memory usage message', is current for the node indicated at the point your job exceeded its memory allocation.
What should I do?
As a first step you can increase your memory request to accommodate your memory usage.
Remember that if you ask for significantly more memory than required you may slow down your job starting as there must be more free memory to run your job than was necessary.
For jobs using less than a full node of CPUs, note that you may be charged more for your job by asking for more memory. Check the job costs page for information on how job charges are calculated.
If your memory usage looks higher than expected for your jobs, you may wish to look into ways to reduce memory usage of the job either through configuration options, input file options, or changing the code if you are developing your own code.
Note that for applications that allow you to tweak their memory usage through configuration or input file options, using more memory often does not actually result in improved performance. You can experiment with different values to determine the most efficient options for your jobs.
How to create self submitting jobs
Each job, as its last step, sets up the scripts and data for the next job to be run, then it issues a qsub command to run that next job.
The biggest problem with self submitting scripts is that if the jobs themselves are failing therefore finishing immediately, and there is no limiter in the self submission process, the constant flood of job submissions can overload the batch system and perhaps also needlessly tie up a large number of cpus if the jobs are parallel.
The simple examples linked below illustrate how to build in a limit to the number of times a job will resubmit itself.
The examples allow the execution of a sequence of checkpoint-restart jobs. These scripts are only templates – you will need to modify them for your application. In particular the PBS qsub options (#PBS lines) will almost definitely need modifying. The executable line and possibly extra processing needed for restarting will have to be added and you may want to modify or eliminate the verbose output the scripts produce.
- undertime.tcsh and undertime.tcsh assume that the compute phase of the job will complete within the walltime of the PBS job. Most likely the compute executable will finish by writing a checkpoint file and doing whatever cleanup is necessary for the follow-on job to start.
- overtime.bash and overtime.tcsh assume that the compute phase of the job may be terminated by PBS due to exceeding the jobs walltime limit. The assumption is that the executable is regularly writing checkpoint files and can handle being terminated at an arbitrary point in execution.
^M: bad interpreter message
If you encounter the message:
/bin/bash^M: bad interpreter: No such file or directory
and you created your batch job script on a Windows host and then copied it to Gadi, you will need to change the format of the line endings in your job script from the Windows standard (CRLF) to the Unix standard (LF).
Job exceeded size limit of stdout and stderr
If your non-interactive job exceeds the combined size limit of 1 GiB for its standard output and error streams, it will be killed, and you may receive a message like the following in the job's '.e' file:
Job 1234.gadi-pbs has exceeded size limit of stdout and stderr on node gadi-cpu-clx-0001. Limit: 1.0GB, Used: 1.23GB. Job killed!
Why is there a size limit for stdout and stderr streams?
For a non-interactive job, PBS spools the job's stdout and stderr to a separate space in the allocated node's local hard disk.
There's limited space on the local hard disk and this needs to be shared between all the jobs running on the node. If a job takes up all the allocated space for this, no more content will be able to be captured.
It also means that other jobs won't be able to run on this node until a manual clean up is done. Hence, a combined size limit of 1GiB has been set for a job's stdout and stderr streams.
What should I do?
For most jobs, a limit of 1 GiB for the standard output and error streams is more than sufficient. If a job uses more than this limit, it is likely to be unintentional. An easy way to fix this is by modifying your job script to redirect the output of your program to a file in your /scratch space, for instance.
Example:
# Redirect stdout and stderr to two separate files
./myprogram > /scratch/proj1/abc123/prog.out.${PBS_JOBID} 2> /scratch/proj1/abc123/prog.err.${PBS_JOBID}
# Combine stdout and stderr and redirect to a file
./myprogram &> /scratch/proj1/abc123/prog.out.${PBS_JOBID}
Authors: Andrew Wellington, Anish Varghese, Mohsin Ali, Tegan Kahn, Yue Sun
Overview
Content Tools