This page shows how to configure training and inference processes for CORDEX-CMIP5 data stored in NCI. The CORDEX-CMIP5 data is described further on our NCI CMIP pages.

You need to join project dk92 to access dl4ds environment and project rr3 to access the CORDEX-CMIP5 dataset.  

CORDEX Data Example

Please clone the example notebook DL4DS_cordex_NCI.ipynb to Gadi. In this example, we have used some CORDEX-CMIP5 data.  While any of the data should work, in this example we have used the following directory:

/g/data/rr3/publications/CORDEX/output/AUS-44/UNSW/CSIRO-BOM-ACCESS1-0/rcp45/r1i1p1/UNSW-WRF360K/v1/day/tasmax/files/d20210629/

This directory contains maximum temperature data prediction from the year 2006 to 2100. The data is divided into three separate parts and used for the training, testing, and validation process. From each subset data, an array of coarse data is created and then both the high-resolution and coarse data are fed to the DL4DS model. A set of example data sets is shown in Figure 1. 


On the left-hand side of Figure 1, the coarse data is shown which is created from the ground truth data shown in the middle. Then the coarse and ground truth data are used to train the model. The right-hand side shows the high-resolution output from the trained model. Note that the train and test data are completely separate and there is no overlap. 

Figure 1. From left to right: Coarsed, ground truth, and predicted data.

Configuration 

The DL4DS is highly configurable, it provides APIs for setting various learning parameters. Figure 2 shows an example of the configuration through API. On the left-hand side, the model configuration API is shown. First, the architectural parameters are set. Then, a supervised trainer is set up with learning parameters. Also, the "GPU" is set at the training device; thus, the trainer will use all the GPUs available to the node automatically. On the right-hand side, model parameters are printed out according to the configuration. 

Figure 2. Left: DL4DS learning configuration. Right: Model parameters printout. 

ARE Notebook

To launch the instance on a GPU, follow the steps: 

  1. Go to https://are.nci.org.au/ and log in. 
  2. In the JupyterLab launch Dashboard, use the following information:
    Walltime (hours):4
    Queue: gpuvolta
    Compute Size: 1gpu
    Project: your own project (such as ab123).
    Storage: gdata/dk92+gdata/rr3+scratch/ab123(change ab123 to your own project id)
  3. In the Advanced options section: 
    Module directories: /g/data/dk92/apps/Modules/modulefiles 
    Modules: dl4ds/1.8.0

  4. Open the notebook "DL4DS_cordex_NCI.ipynb" which is cloned from https://github.com/nci/DL4DS-nci

Inference Results for the CORDEX Data: 

Figure 3 shows the ground truth and prediction of the DL4DS model for the NCI CORDEX data. The video shows how a prediction and the corresponding ground truth can be selected using the slider. For each time position, the left-hand side shows ground truth and the right-hand side shows the corresponding predicted value. 

Figure 3. Ground truth and prediction for the NCI CORDEX dataset. 

































































































































































  • No labels