Gadi_jupyterlab

Gadi_jupyterlab module aims to help users to set up several pre-defined services like JupyterLab, Dask cluster and Ray cluster utilising Gadi resource. Its functionalities are listed below with alternative solutions available.

It could be used to set up the pre-defined Dask and Ray cluster across multiple CPU and GPU nodes. 

gadi_jupyterlab

Client(dask.LocalCluster)

Client(scheduler_file=os.environ["DASK_PBS_SCHEDULER"])

dask_jobqueue.PBSCluster()

Client(dask_cuda.LocalCUDACluster)

Client(scheduler_file=os.environ["DASK_PBS_SCHEDULER"])

                       ray.init()

ray.init(address="auto")

                        ray.init()

ray.init(address="auto")

The gadi_jupyterlab module contains several bash scripts and you can access them by loading the gadi_jupyterlab module under dk92 project ( Please request to join dk92 if you are not the member yet).

$ module use /g/data/dk92/apps/Modules/modulefiles
$ module load gadi_jupyterlab/22.06

The gadi_jupyter module is designed to work with external Python environments so it doesn't contain any python library. You must specify another python environment which may contains jupyter, dask or Ray libraries. For example, you ca load the "NCI-data-analysis/2022.09" as below

$ module use /g/data/dk92/apps/Modules/modulefiles
$ module load NCI-data-analysis/2022.06
$ module load gadi_jupyterlab/22.06

For newer Dask version from 2023.02, please load the module gadi_jupyterlab/23.02

$ module use /g/data/dk92/apps/Modules/modulefiles
$ module load NCI-data-analysis/2023.02
$ module load gadi_jupyterlab/23.02

Please note: you are recommended to use ARE Jupyterlab app to set up a Jupyterlab session at Gadi. You can still use the gadi_jupyterlab module to set up the pre-defined Dask or Ray clusters in ARE Jupyterlab app. 

Now you can run the  gadi_jupyterlab scripts to set up the following services in your Gadi PBS job.