Under ARE, the JupyterLab App allows you to start a JupyterLab instance dedicated for your use.
Starting a Session
- Login to ARE: https://are.nci.org.au/
- Click the JupyterLab icon on the Dashboard (home page)
- the number of hours you want your session to run for (maximum)
- the compute queue you want to submit the job to
- the size of the compute resources you have access to (maximum). If you attempt to exceed the CPU limit your processes will be throttled to the amount requested. If you exceed the memory your Jupyter Kernel may be terminated. Note: this is the resources used within the notebook; if you offload the processing to other jobs (e.g. using Dask) then you may not need a very large JupyterLab session
- select which project to allocate the SU from (must be a project with a current allocation, or the session will not start)
- select what storage (gdata and scratch areas) are required for your job to run
- select additional software licenses if required for your job
- if you want to receive an email once your session has started. This is useful when the cluster is busy and your session is queued waiting for a free slot
- Optionally use the advanced options to specify an alternative JupyterLab installation or other module files that your notebooks use
- Module directories: a space-separated list of additional directories to search for modulefile definitions. If these are under /g/data or /scratch make sure you've included the appropriate directive(s) in your storage options above.
- Modules: a space-separated list of modules to be loaded
- Python or Conda virtual environment base: specify the top-level directory of a custom virtual env, or the base directory of a conda environment. If this is under /g/data or /scratch make sure you've included the appropriate directive in your storage options above.
- Conda environment: specify the name of a conda environment to activate. You must supply a conda base directory in order for this to work.
- Additional environment variables needed for your job
- jobfs size allocated to your job
- Additional PBS flags not covered by other options
- A pre-script to be run in your job before JupyterLab starts
- Click Launch button
- Initially it will show the status as Queued until your session begins to start. Once it is finished it will change to Running and look like the figure below
- Click the Open JupyterLab button
- This will open a new tab in your web browser that shows the JupyterLab session. NOTE: it can take a minute or two for the JupyterLab Client to start
- If you close the window the session will remain running. To end the session you can either Shutdown (last item in File menu) or delete the job on the Interactive Sessions page of ARE
Note for Dask usage: You can use the JupyterLab session resources as above to set up a local Dask cluster within a Jupyter notebook. If you need a larger scale Dask cluster, please select Small or Medium sized Compute Size to start up the JupyterLab session and then set up another Dask cluster crossing multiple nodes within a Jupyter notebook.
Loading Linux Environment modules
To load in specific modules you already need to know where these are located on the system by either discovery under a VDI session, or someone has shared with you.
1. Loading from /apps
Click "Advanced options" and type in "<module name>" in "Modules" box and leave "Module directories" box empty. You can specify multiple modules by including a space between each.
For example you can load modules python3/3.7.4 and python-shapely/1.7.0-py3.7 as below:
2. Loading modules from other locations
Click "advanced options" and
1. type in "<your module directory>" in "Module directories" box. If this is under /g/data or /scratch, make sure you include the appropriate directive in the storage box on the form.
2. type in "<module name>" in "Modules" box
For example, we can load the "NCI-data-analysis/2021.03" module from "/g/data/dk92/apps" as below: