After starting a PB job, we can invoke jupyter.ini.sh script to set up a pre-defined Ray cluster to utilise the compute resources requested by the job. 

Multiple CPU nodes

You can specify the option "-R" to set up a Ray cluster with the requested PBS resources i.e. using each CPU core as a single threaded Ray worker. In this case, you can specify the number of Ray workers per node by specifying the "-p" options. 

...

Note a Ray worker always contains 1 thread so "-t" is invalid in setting up the Ray cluster.

Multiple GPU nodes

You can also specify flag "-Gg" together with "-R" when running jupyter.ini.sh to set up a Rask cluster by using Gadi GPU devices. As default, the number of Dask workers equals the number of GPU devices requested in the PBS job and each worker has 1 thread. 

...