How to use
Typically, running a Matlab job on Gadi that benefits from the speedup of multi-cores would require,
- joining the matlab software group for your institution,
matlab_<institution>
, to allow your access to the Matlab licence - load modulefile for the corresponding licence
- setting up both your
.m
file and the job submission script to allow the computation- and data-intensive problems using multi-cores, GPUs and multi-nodes. - Make use of the smpd deamon available in Matlab for parallelisation.
Warning |
---|
When using smpd, please specify -smpdfile to $PBS_JOBFS/<file> to avoid creating smpd files in /tmp. |
Regarding the licence access, please check the section Licence
below for more details.
Warning |
---|
If you know your Matlab application is neither computation nor data intensive, please see the advice in the section 'Running Single-core Jobs'. |
Once get the access to the matlab licence on Gadi, please confirm with your institution that whether the licence comes with either of the features 'Parallel Computing Toolbox' and 'Distributed Computing Server'. Although some linear algebra and numerical functions, such as matrix products, fft
, eig
, sort
, are implicitly multi-threaded and make use of all the cores available to the job without any extra programming efforts, see an incomplete list provided by Mathworks Support Team here, the majority of the functions provided by matlab runs only on a single core and users need to explicitly use the functionalities provided by these two features to enable the utilisation of multi-cores and multi-compute nodes, respectively. As of 30 Oct 2020, the majority of NCI-hosted matlab licences provides only the feature Parallel Computing Toolbox
, therefore, supports jobs running only within a single compute node.
In order to utilise multi-cores within a single compute node, typically, the minimum programming effort would be the revision of the for
loops into parfor
loops in the .m
file. For example, to parallelise the matlab jobs with the majority computation in a for loop read as
Code Block |
---|
|
% Initialisation
A = 512;
N = 1024;
y = NaN(1,N);
% Executing serial `for` loop
for i = 1:N
y(i) = max(abs(eig(rand(A))))
end
% Print results
disp(y);
disp("Done"); |
simply do
Code Block |
---|
|
% Initialisation
numberOfWorkers = 48;
A = 512;
N = 1024;
y = NaN(1,N);
% Parallel pool creation
parpool('local', numberOfWorkers);
% Executing parallel `for` loop
parfor(i = 1:N, numberOfWorkers)
y(i) = max(abs(eig(rand(A))))
end
% Print results
disp(y);
disp("Done"); |
For further explorations of the functionality provided by the Parallel Computing Toolbox
, please refer to Mathworks website. If you want to run Matlab jobs across more than one compute nodes and confirm that you get access to the feature Distributed Computing Server
, please contact help@nci.org.au for further instructions.
Once the .m
file is ready, you will need to load the MATLAB application and licence module.
To load the specific version of R2019b, please run
Code Block |
---|
|
$ module load matlab/R2019b |
Warning |
---|
It is always better to load the module with a specific version in order to make your job replicable because the default version changes over time. |
To load the licence module for your own matlab software group, please run
Code Block |
---|
|
$ module load matlab_licence/<institution> |
An example job submission script matlabjob.sh
is provided below.
It requests 48CPUs, 190 GiB of memory, 400GiB of local disk space for 2 hours in the normal queue. Within the job it passes the number of CPUs as the number of matlabWorkers and the PBS_JOBFS folder as the output directory outputDir,
both of which are variables used in the matlab script mfile.m
.
Note that the submission script uses the PBS directive`-l software`
, to ask the job scheduler for the reservation of matlab token seats from licence matlab_<institution>
where the placeholder <institution>
needs to be replaced by your own institution, see details in the section Licence requirements below. To change the number of processors, memory, or jobfs required, and/or project folders to mount, simply modify the appropriate PBS resource requests at the top of this file.
Code Block |
---|
|
#!/bin/bash
#PBS -q normal
#PBS -l walltime=02:00:00
#PBS -l ncpus=48
#PBS -l mem=190GB
#PBS -l jobfs=400GB
#PBS -l software=matlab_<institution>
#PBS -l wd
# Load modules, always specify version number.
module load matlab/R2019b
module load matlab_licence/<institution>
# Must include `#PBS -l storage=scratch/ab12+gdata/yz98` if the job
# needs access to `/scratch/ab12/` and `/g/data/yz98/`
# Run matlab application
matlab -nodisplay -nosplash -r "outputDir='$PBS_JOBFS',numberOfWorkers=$PBS_NCPUS, mfile, quit" > $PBS_JOBID.log |
Please note, the flag -nodisplay
passed to matlab
enables running without the GUI, while the flag -nosplash
disables the display of the Matlab logo.
Once the job submission script is saved to the same directory in which the matlab script mfile.m
is located, run
Code Block |
---|
|
$ qsub matlabjob.sh |
inside that directory to submit your job to the PBS scheduler.
Running Single-core Jobs Anchor |
---|
| Running Single-core Jobs |
---|
| Running Single-core Jobs |
---|
|
Since Matlab is automatically multithreaded and, by default, it generates as many matlabWorkers as it is allowed to, it is always safe to explicitly set the maximum number of computational threads to one by adding the following line
Code Block |
---|
|
maxNumCompThreads(1) |
at the beginning of the .m
file or starting matlab with the option -singleCompThread
as shown in the following example submission script.
Code Block |
---|
|
#!/bin/bash
#PBS -q normal
#PBS -l ncpus=1
#PBS -l walltime=00:02:00
#PBS -l mem=250MB
#PBS -l software=matlab_<institution>
#PBS -l wd
# Load modules, always specify version number.
module load matlab/R2019b
module load matlab_licence/<institution>
# Must include `#PBS -l storage=scratch/ab12+gdata/yz98` if the job
# needs access to `/scratch/ab12/` and `/g/data/yz98/`
# Run matlab application
matlab -nosplash -nojvm -singleCompThread < mfile.m > $PBS_JOBID.log |
Licence requirements Anchor |
---|
| License requirements |
---|
| License requirements |
---|
|
NCI has a Hosting Provider Agreement with MathWorks that allows us to install MATLAB on our systems and have users provide their own licence to access it. In particular, if your institution has a licence for MATLAB that you can use on the computer in your office then it is most likely possible for you to use it on NCI systems. As of 22 Oct. 2021, there are 14 institutions have set up their configurations with NCI. If it’s not already set up, contact your licence administrator to ask them to get in touch with us.
To get access to your own institution provided Matlab licence, join the corresponding software group in the table below on my.nci.org.au first. Once your group membership request is approved, the corresponding licence module becomes visible to use. Run `module avail matlab` to see the licence modulefile name. Jobs need to load the licence module together with the application module to run Matlab.
Institution | PBS Directive `-lsoftware=` | software group |
---|
The Australian National University | matlab_anu | matlab_anu |
CSIRO* | matlab_csiro | matlab_csiro |
Flinders University | matlab_flinders | matlab_flinders |
The University of Newcastle | matlab_newcastle | matlab_newcastle |
Latrobe University | matlab_latrobe | matlab_latrobe |
Macquarie University | matlab_macquarie | matlab_mq |
The University of Melbourne | matlab_melbourne | matlab_melbourne |
Monash University | matlab_monash | matlab_monash |
Royal Melbourne Institute of Technology | matlab_rmit | matlab_rmit |
The University of New South Wales | matlab_unsw | matlab_unsw |
The University of Wollongong | matlab_wollongong | matlab_wollongong |
The University of Sydney | matlab_usyd | matlab_usyd |
Victoria University | matlab_vu | matlab_vu |
University of South Australia | matlab_unisa | matlab_unisa |
University of Tasmania | matlab_utas | matlab_utas |
* CSIRO has a special arrangement with NCI that allows only a specific group of users to use their licence. Please contact CSIRO licencing service to discuss the access. We can only add users to the group on their approval.