MPI...

How to use 

Both Open MPI and Intel MPI are supported on Gadi.

You can check the versions installed in Gadi with a module query:

 $ module avail openmpi

or

 $ module avail intel-mpi

We normally recommend using the latest version available and always recommend to specify the version number with the module command:

 $ module load openmpi/4.1.1

or

 $ module load intel-mpi/2021.3.0

For more details on using modules see our software applications guide.

Compiling and Linking

Follow the link https://opus.nci.org.au/display/Help/Compiling+and+Linking#CompilingandLinking-UsingMPI for information about how to create a binary executable of your MPI-enabled application using MPI libraries.

Open MPI Job 

An example PBS job submission script named mpi_job.sh is provided below.

It requests 48 CPUs, 128 GiB memory, and 400 GiB local disk on a compute node on Gadi from the normal queue for 30 minutes against the project a00. It also requests the system to enter the working directory once the job is started. This script should be saved in the working directory from which the analysis will be done.

To change the number of CPU cores, memory, or jobfs required, simply modify the appropriate PBS resource requests at the top of the job script files according to the information available in our queue structure guide. 

Note that if your application does not work in parallel, setting the number of CPU cores to 1 and changing the memory and jobfs accordingly is required to prevent the compute resource waste.

#!/bin/bash
 
#PBS -P a00
#PBS -q normal
#PBS -l ncpus=48
#PBS -l mem=128GB
#PBS -l jobfs=400GB
#PBS -l walltime=00:30:00
#PBS -l wd
 
# Load module, always specify version number.
module load openmpi/4.1.1
 
# Must include `#PBS -l storage=scratch/ab12+gdata/yz98` if the job
# needs access to `/scratch/ab12/` and `/g/data/yz98/`
 
# Run application
mpirun -np $PBS_NCPUS <your MPI exe>

Intel MPI Job 

An example PBS job submission script named mpi_job.sh is provided below.

It requests 48 CPU s, 128 GiB memory, and 400 GiB local disk on a compute node on Gadi from the normal queue for 30 minutes against the project a00. It also requests the system to enter the working directory once the job is started. This script should be saved in the working directory from which the analysis will be done.

To change the number of CPU cores, memory, or jobfs required, simply modify the appropriate PBS resource requests at the top of the job script files according to the information available in our queue structure guide.

 Note that if your application does not work in parallel, setting the number of CPU cores to 1 and changing the memory and jobfs accordingly is required to prevent the compute resource waste.

#!/bin/bash
 
#PBS -P a00
#PBS -q normal
#PBS -l ncpus=48
#PBS -l mem=128GB
#PBS -l jobfs=400GB
#PBS -l walltime=00:30:00
#PBS -l wd
 
# Load module, always specify version number.
module load intel-mpi/2021.3.0
 
# Must include `#PBS -l storage=scratch/ab12+gdata/yz98` if the job
# needs access to `/scratch/ab12/` and `/g/data/yz98/`
 
export I_MPI_HYDRA_BRANCH_COUNT=$(($PBS_NCPUS / $PBS_NCI_NCPUS_PER_NODE))
 
# Run application
mpirun -np $PBS_NCPUS <your MPI exe>

The I_MPI_HYDRA_BRANCH_COUNT variable sets the number of nodes for the Intel MPI.

Submit and Run Job

To submit and run the job you would use the PBS command:

$ qsub mpi_job.sh

Hybrid MPI and OpenMP 

Many modern program can run in so called hybrid mode, i.e. an MPI communication is done between the nodes (or physical CPUs) and OMP threads are used inside nodes (or physical CPUs). This may allow you to get a better performance. If you have a program that support such a hybrid mode, please  reference this guide and follow the link below for more information:

• https://opus.nci.org.au/display/Help/Compiling+and+Linking#CompilingandLinking-UsingHybridMPIandOpenMP

Authors: Mohsin Ali