Specialised Environments

Page tree

Introduction

FourCastNeXt is an optimization of the original FourCastNet library.  In FourCastNext, we show improved methods that can train FourCastNet using only 1% compute required by the baseline, while maintaining model performance on par, or even better, than the baseline.

Code and Data Location

The datasets and pre-trained models for FourCastNeXt are available at /g/data/wb00/FourCastNet (see https://dx.doi.org/10.25914/pptj-g181 on how to connect). A description of the dataset area (/g/data/wb00/FourCastNet) is provided below:

The example Jupyter Notebook may be found on the Github repository: https://github.com/nci/FourCastNeXt.git. The pre-trained model for FourCastNeXt is available at /g/data/wb00/FourCastNet/nci/pretrained/ . The prediction results using FourCastNeXt are available at /g/data/wb00/FourCastNet/nci/predictions/FourCastNeXt.

Inference Notebook

We have created a FourCastNext inference notebook that aims to analyse the predictions generated using FourCastNext and compare the error rates with those of FourCastNet.

You need to clone the FourCastNeXt repository to your own working space on Gadi. For example: If you are working on /scratch/ab12, then follow the given commands to clone FourCastNeXt.

cd /scratch/ab12/your_user_id
git clone https://github.com/nci/FourCastNeXt.git

Then please follow steps in "Start ARE JupyterLab Session" to run the inference notebook. 

Tutorial Notebook 

For those who want to grasp the fundementals of the FourCastNeXt model, the full-fledged model is complex that involves setting up the distributed training and data loading clusters. With simplicity mind, we have developed a simplified version of the training and the model code that can run entirely from a Jupyter notebook: https://github.com/NCI-HPCDI/BOM-ML-workshop

You need to clone the FourCastNeXt tutorial repository to your own working space on Gadi. For example: If you are working on /scratch/ab12, then follow the given commands to clone FourCastNeXt tutorial.

cd /scratch/ab12/your_user_id
git clone https://github.com/NCI-HPCDI/BOM-ML-workshop.git

Then please follow steps in "Start ARE JupyterLab Session" to run the tutorial notebook. 

Start ARE JupyterLab Session 

Launch an ARE Instance

  1. Go to https://are.nci.org.au/ and login. 
  2. Click on the Jupyter notebook app and a form will appear.
  3. In the Dashboard for JupyterLab launch, fill up the followings.
    Walltime (hours): 10 ( You could request shorter Walltime to run tutorial notebook)
    Queue: gpuvolta
    Compute Size: 1gpu
    Project: your own project (example in the figure below: ab12)
    Storage: gdata/dk92+gdata/wb00+gdata/rt52+scratch/ab12 (i.e. your working space storage)

  4. In the Advanced options section: 
    Module Directories: /g/data/dk92/apps/Modules/modulefiles
    Modules: NCI-ai-ml/23.10

  5. Now, Launch the JupyterLab Instance.

Launch the Notebook

Inference notebook

  1. In the jupyter notebook, open the notebook by navigating to the cloned repository, i.e. FourCastNeXt/notebooks/ and opening inference.ipynb.
  2. In the setting up section, remember to change the current directory by setting path in DIR = "/CURRENT/WORKING/DIRECTORY/". 
  3. All the other code is already present and there is no need to write anything. If you want to run the cells, remember to start from top.

Tutorial notebook

  1. In the jupyter notebook, open the notebook by navigating to the cloned repository, i.e. BOM-ML-workshop and opening notebook.ipynb.
  2. All the other code is already present and there is no need to write anything. If you want to run the cells, remember to start from top.
  • No labels