You can set up a Ray cluster to utilise single node or multiple nodes resources.

Single Node

If you are working in a with single node resources via OOD, ARE or Gadi PBS job, you can simply callcall ray.init() within  in your Jupyter Notebook or python script to start a Ray runtime on the current working host.

...

An easy way to set up the pre-defined Ray cluster is to utilise dk92 module "gadi_jupyterlab".  

Launching a

...

pre-defined Ray cluster

Gadi

In your PBS job script, you should load NCI-data-analysis/2022.06 together with the gadi_jupyterlab module. Then you need to run a script called "jupyter.ini.sh -R" to set up the pre-defined Ray cluster. It will start a Ray worker on each CPU core of all available compute nodes in a job. You can also specify the number of Ray workers per node via "-p" flag. For example, in a job requesting 96 cores ( 2 nodes) of "normal" queue, you can set up a pre-defined Ray cluster with 12 Ray workers per node, and 24 Ray workers in total via the following command

...

First of all, you need to request multiple nodes in ARE JupyterLab session and specify the proper storage projects.

Image Removed

...

nodes  and start a JupyterLab session as below

...

...

...

...

...

Image Removed

...

...

In the Jupyter notebook, using the following lines to connect the pre-defined Ray cluster and print the resources information. 

import ray
ray.init(address="auto")
print(ray.cluster_resources())

You will see 96 CPU Cores and two nodes are used by the cluster as expected.

...