Overview
On Gadi, we strongly recommend using nci-parallel.
You need to have a very convincing case if you are using GNU parallel instead of nci-parallel.
The examples below are here mostly for historical reasons.
GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel.
How to use
Example script below shows how to run many single CPU tasks that may need (very) different time to execute.
Many thanks to one of our users, Scott Wales, for sharing this example.
#!/bin/bash
# Run an embarrassingly parallel job, where each command is totally independent
# Uses gnu parallel as a task scheduler, then executes each task on the available cpus with pbsdsh
#PBS -q normal
#PBS -l ncpus=480
#PBS -l walltime=48:00:00
#PBS -l mem=1000gb
#PBS -l wd
# Load module, always specify version number.
module load parallel/20191022
# Must include `#PBS -l storage=scratch/ab12+gdata/yz98` if the job
# needs access to `/scratch/ab12/` and `/g/data/yz98/`
SCRIPT=$PWD/script.sh # Script to run.
INPUTS=inputs.txt # Each line in this file is used as arguments to ${SCRIPT}
# It's fine to have more input lines than you have requested cpus,
# extra jobs will be executed as cpus become available
# Here '{%}' gets replaced with the job slot ({1..$PBS_NCPUS})
# and '{}' gets replaced with a line from ${INPUTS}.
# Pbsdsh starts a very minimal shell. `bash -l` loads all of your startup files, so that things like modules work.
# The `-c` is so that bash separates out the arguments correctly (otherwise they're all in a single string)
parallel -j ${PBS_NCPUS} pbsdsh -n {%} -- bash -l -c '"'${SCRIPT} {}'"' :::: ${INPUTS}
# Note that CSH needs the following line:
#parallel -j ${PBS_NCPUS} pbsdsh -n {%} -- bash -l -c '"${SCRIPT} {}"' :::: ${INPUTS}
When using the above example it is important to select an appropriate number of CPUs vs. number of tasks that needs to be run. (A number of tasks in the example is equal to the number of lines in input file INPUTS). As a rough guide, select the number of cpus to be about 10 times smaller than the number of tasks. Example above uses 480 CPUs, this should be OK for 4000-10000 tasks.
Other useful parallel lines:
1) To run commands in the input.cmd file:
$ cat input.cmd | parallel -j ${PBS_NCPUS} pbsdsh -n {%} -- bash -l -c '{}'
2) To run only one command on a node
$ nt=${PBS_NCI_NCPUS_PER_NODE}
$ parallel -j $((${PBS_NCPUS}/${nt})) --rpl '{%} 1 $_=($job->slot()-1)'"*${nt}" pbsdsh -n {%} -- bash -l -c '"'${SCRIPT} {}'"' :::: ${INPUTS}
3) To give nt threads to each command on a node (in the below example, nt=16):
$ nt=16
$ parallel -j $((${PBS_NCPUS}/${nt})) --rpl '{%} 1 $_=($job->slot()-1)'"*${nt}" pbsdsh -n {%} -- bash -l -c '"'${SCRIPT} {}'"' :::: ${INPUTS}
Each line in the INPUTS is $1 argument in SCRIPT.
Authors: Andrey Bliznyuk, Mohsin Ali
Overview
Content Tools