Introduction

Our NCI AI/ML environment under project dk92 has been developed to allow Machine Learning analysis and processing by utilising Gadi GPU resources. The module contains popular machine learning packages including tensorflow-gpu and pytorch-gpu together with general purposed data processing packages such as xarray, dask and ray. You can utilise multiple GPU nodes to run Tensorflow or Pytorch based projects via parallelisation frameworks like Horovod and Ray. You can access our NCI AL/ML environment by loading the 'NCI-ai-ml' module under project dk92.

The AI/ML packages are available in two ways on Gadi: natively compiled standalone packages under /apps, or using the NCI-ai-ml module described here. In our module we have bundled together the major data science and machine learning packages (including Pytorch, Tensorflow, and Horovod) and configured them so that they are interoperable. The standalone packages can be difficult to use when trying multiple technologies. Our experiments show that the performance difference between the two methods is less than 5% in most cases.

Accessing the module

Resources can be accessed by joining the project dk92. Note that no storage or compute resources are provided by project dk92 as it is purely for accessing the software. You will need to use your existing compute NCI project code for computational resources.

Package components

The NCI-ai-ml environment is regularly updated and the latest version is 24.08. Software packages are available for Python and the major ones are listed as below

Parallelisation frameworks

Currently the NCI-ai-ml environment supports the following parallelism ML frameworks:

• Horovod
• Ray
• torchrun
• Pytorch-lightning

Version history