If you require more CPU/GPU resources than what is available interactively on the ARE VDI, there is the option to launch a Paraview server (pvserver) from within a Gadi batch submission compute job, and connect to this server from your local installation of Paraview. Note that in order for this to work, your local version of Paraview must be the same as the version on Gadi.
To see which versions of ParaView are currently available on Gadi:
$ module avail paraview ------------------------------------------------------------ /apps/Modules/modulefiles ------------------------------------------------------------- paraview/5.8.0 paraview/5.8.0-mesa paraview/5.9.1-gpu paraview/5.10.1 paraview/5.8.0-gpu paraview/5.9.1 paraview/5.9.1-mesa paraview/5.10.1-mesa
The three types of ParaView modules are:
- paraview/<version> which was built with the opengl library. This can be used with the X11 server to do rendering on Gadi.
- paraview/<version>-mesa which was built with the mesa library.
- paraview/<version>-gpu which was built with the NVIDIA EGL and VISTRX libraries. This module needs to be run using the gpuvolta queue.
For the following examples, we will use paraview/5.9.1-mesa and paraview/5.9.1-gpu.
paraview/5.9.1-mesa
First we will create a PBS submission script that we will run interactively on Gadi:
#!/bin/bash #PBS -N paraview-test #PBS -q normal #PBS -P <insert_project_code_to_charge_to> #PBS -l walltime=1:00:00 #PBS -l ncpus=48 #PBS -l mem=190GB #PBS -l jobfs=10GB #PBS -l storage=gdata/<project_code>+gdata/<project_code>
For this example, we are requesting 48 CPUs and 190GB of memory. Now we can save this submission script (e.g., paraview.sh). To submit this as an interactive job:
$ qsub -I paraview.sh qsub: waiting for job 36501851.gadi-pbs to start qsub: job 36501851.gadi-pbs ready
Once the interactive job is ready, load in the paraview-mesa module:
$ module load paraview/5.9.1-mesa
Next, let's start our ParaView server using mpirun on the 48 processes we requested:
$ mpirun -np 48 pvserver &
Once the pvserver has launched, it will print a connection message:
Waiting for client... Connection URL: cs://gadi-cpu-clx-1467.gadi.nci.org.au:11111 Accepting connection(s): gadi-cpu-clx-1467.gadi.nci.org.au:11111
Next you will need to open up a new terminal on your local machine and create an ssh tunnel through the Gadi login node:
ssh -L11111:gadi-cpu-clx-1467.gadi.nci.org.au:11111 <your_NCI_user_name>@gadi.nci.org.au
Finally, you will need to launch your local version of ParaView (that matches the version on Gadi) and go to File → Connect which will open up the Server Configuration menu:
Click on Add Server and give it a Name (keep Host as localhost and Port as 11111). Then click Configure and Save. Now click the Connect button:
If you now look in your Pipeline Browser, you will see you are connected to Gadi:
Go to View → Memory Inspector to confirm you are connected to the 48 processes that you requested:
If you go to your interactive job terminal and run htop, you can see the pvserver CPUs being utilised:
Next we can try to run some Paraview processes and monitor the CPUs being used by our pvserver.
First let's start a Python trace (Tools → Start Trace) so we can record what we are doing for later tests. Now let's create a wavelet source by running Source → Search and type in Wavelet
Now go to the Properties tab and increase the "Whole Extent" to [-400,400; -400,400; -400,400], change the coloring to "RTData", change the Representation to "Surface" and click "Apply"
If we click the Information tab, we can see that this Image contains 512 Million Cells and ~519 Million points. Let's now try and run some filters on this image.
First we can try a "Contour" filter (Filters → Search and type in Contour) on our Wavelet:
Let's go to the properties tab and change the Coloring to "Normals" and select our favorite Colour preset:
Next let's run a Clip filter on our Wavelet and select "Clip Type" as "Scalar", uncheck the "Invert" box and click "Apply":
When running these filters, you should be able to see your pvserver CPUs doing some work:
If we look at the properties of our Contour, we can see it's a Polygonal Mesh with ~11 Million cells and 5.5 Million points. Our Clip produced an Unstructured Grid with ~268 Million cells, 262 Million points and took approximately 4 times the memory to produce. We can view our Coutour and Clip side by side by creating a Split Render View. For this, we will compare RTData:
You can link the two images by right clicking on one view, and selecting "Link Camera" and linking to the other view. Let's stop our trace (Tools → Stop Trace) and save as a Python file for later use.
paraview/5.9.1-gpu
Setting up Paraview on Gadi to utilise GPUs is similar to our CPU example. First we will need create a PBS submission script that we will run interactively on Gadi:
#!/bin/bash #PBS -N paraview-test #PBS -q gpuvolta #PBS -P <insert_project_code_to_charge_to> #PBS -l walltime=1:00:00 #PBS -l ncpus=48 #PBS -l ngpus=4 #PBS -l mem=190GB #PBS -l jobfs=10GB #PBS -l storage=gdata/<project_code>+gdata/<project_code>
Note that you will need to use the gpuvolta queue to run Paraview with GPUs. For this example, we are requesting 4 GPUs, 48 CPUs and 190GB of memory. Now we can save this submission script (e.g., paraview_gpu.sh). To submit this as an interactive job:
$ qsub -I paraview_gpu.sh qsub: waiting for job 36688248.gadi-pbs to start qsub: job 36688248.gadi-pbs ready
Once the job is ready, load in the Paraview GPU module:
$ module load paraview/5.9.1-gpu
To monitor the GPUs that were requested:
$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.82.01 Driver Version: 470.82.01 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... On | 00000000:3D:00.0 Off | 0 |
| N/A 34C P0 44W / 300W | 0MiB / 32510MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-SXM2... On | 00000000:3E:00.0 Off | 0 |
| N/A 32C P0 41W / 300W | 0MiB / 32510MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 2 Tesla V100-SXM2... On | 00000000:B1:00.0 Off | 0 |
| N/A 32C P0 42W / 300W | 0MiB / 32510MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 3 Tesla V100-SXM2... On | 00000000:B2:00.0 Off | 0 |
| N/A 33C P0 42W / 300W | 0MiB / 32510MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
To start our Paraview server using mpirun with the 4GPUs and 48 CPUs we requested:
$ mpirun --map-by node -np 12 pvserver --egl-device-index=0 --force-offscreen-rendering : -np 12 pvserver --egl-device-index=1 --force-offscreen-rendering : -np 12 pvserver --egl-device-index=2 --force-offscreen-rendering : -np 12 pvserver --egl-device-index=3 --force-offscreen-rendering &
Here we are assigning 12 CPUs per GPU, where the--egl-device-index= flag refers to the GPU number. Once pvserver has launched, it will print a connection message:
Connection URL: cs://gadi-gpu-v100-0114.gadi.nci.org.au:11111 Accepting connection(s): gadi-gpu-v100-0114.gadi.nci.org.au:11111
Next you will need to open up a new terminal on your local machine and create an ssh tunnel through the Gadi login node:
ssh -L11111:gadi-gpu-v100-0114.gadi.nci.org.au:11111 <your_NCI_user_name>@gadi.nci.org.au
As in the previous example, you will need to launch your local version of Paraview (that matches the version on Gadi) and go to File → Connect and connect to Gadi. Now let's run the Python script we created in the previous example:
To monitor our GPUs when moving our RenderViews, go to your interactive terminal and run:
$ watch -n0.1 nvidia-smi +-----------------------------------------------------------------------------+ | NVIDIA-SMI 470.82.01 Driver Version: 470.82.01 CUDA Version: 11.4 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Tesla V100-SXM2... On | 00000000:3D:00.0 Off | 0 | | N/A 34C P0 44W / 300W | 1023MiB / 32510MiB | 25% Default| | | | N/A | +-------------------------------+----------------------+----------------------+ | 1 Tesla V100-SXM2... On | 00000000:3E:00.0 Off | 0 | | N/A 32C P0 41W / 300W | 773MiB / 32510MiB | 22% Default| | | | N/A | +-------------------------------+----------------------+----------------------+ | 2 Tesla V100-SXM2... On | 00000000:B1:00.0 Off | 0 | | N/A 32C P0 42W / 300W | 788MiB / 32510MiB | 18% Default| | | | N/A | +-------------------------------+----------------------+----------------------+ | 3 Tesla V100-SXM2... On | 00000000:B2:00.0 Off | 0 | | N/A 33C P0 42W / 300W | 773MiB / 32510MiB | 28% Default| | | | N/A |
As you move the Paraview RenderViews, you will notice the Volatile GPU-Util increasing for all GPUs.