Many users will require a JupyterLab environment with packages included beyond those available in the system default python environment.
Some of those needs may be satisfied by using one of the Specialised Environments but those won't be appropriate for all users.
If you need a custom environment, here is a step-by-step guide on getting started.
You will need to substitute appropriate values where I have used the ne4 project and my userid of dmo900 throughout; you may also not want to install the environment under /g/data/$PROJECT/$USERID (for example you might want it in /scratch if it is only for short term usage, or maybe directly under /g/data/$PROJECT if it is for use project-wide rather than just for your own use). Just make sure that you are consistent between the creation of the environment, the path you pass in the "Python or Conda virtual environment base", and ensure your job has the appropriate storage option applied to be able to see that path.
Creating the venv
Log into Gadi, and load your preferred python module:
$ module load python3/3.10.4Loading python3/3.10.4Loading requirement: intel-mkl/2021.4.0$Create a new virtual environment:
$ python3 -m venv /g/data/ne4/dmo900/testvenv$(we suggest putting the venv in /g/data or /scratch, as they can grow too large for storing in your home directory)
Activate the venv in the current session (note the leading "dot-space" to source the file into the current shell environment):
$ . /g/data/ne4/dmo900/testvenv/bin/activate(testvenv) $- Update the pip and setuptools packages:
(testvenv) $ pip install -U pip setuptools
Requirement already satisfied: pip in /g/data/ne4/dmo900/testvenv/lib/python3.10/site-packages (22.0.4)
Collecting pip
Downloading pip-22.2.2-py3-none-any.whl (2.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 28.8 MB/s eta 0:00:00
Requirement already satisfied: setuptools in /g/data/ne4/dmo900/testvenv/lib/python3.10/site-packages (58.1.0)
Collecting setuptools
Downloading setuptools-65.3.0-py3-none-any.whl (1.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 37.2 MB/s eta 0:00:00
Installing collected packages: setuptools, pip
Attempting uninstall: setuptools
Found existing installation: setuptools 58.1.0
Uninstalling setuptools-58.1.0:
Successfully uninstalled setuptools-58.1.0
Attempting uninstall: pip
Found existing installation: pip 22.0.4
Uninstalling pip-22.0.4:
Successfully uninstalled pip-22.0.4
Successfully installed pip-22.2.2 setuptools-65.3.0
(testvenv) $
Install jupyterlab, plus any additional packages you need for example:
(testvenv) $ pip install jupyterlab xarrayCollecting jupyterlabDownloading jupyterlab-3.4.5-py3-none-any.whl (8.8 MB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.8/8.8 MB 63.1 MB/s eta 0:00:00Collecting xarray[...lot of output...]Successfully installed MarkupSafe-2.1.1 Send2Trash-1.8.0 anyio-3.6.1 argon2-cffi-21.3.0 argon2-cffi-bindings-21.2.0 asttokens-2.0.8 attrs-22.1.0 babel-2.10.3 backcall-0.2.0 beautifulsoup4-4.11.1 bleach-5.0.1 certifi-2022.6.15 cffi-1.15.1 charset-normalizer-2.1.1 debugpy-1.6.3 decorator-5.1.1 defusedxml-0.7.1 entrypoints-0.4 executing-1.0.0 fastjsonschema-2.16.1 idna-3.3 ipykernel-6.15.2 ipython-8.4.0 ipython-genutils-0.2.0 jedi-0.18.1 jinja2-3.1.2 json5-0.9.10 jsonschema-4.15.0 jupyter-client-7.3.5 jupyter-core-4.11.1 jupyter-server-1.18.1 jupyterlab-3.4.5 jupyterlab-pygments-0.2.2 jupyterlab-server-2.15.1 lxml-4.9.1 matplotlib-inline-0.1.6 mistune-2.0.4 nbclassic-0.4.3 nbclient-0.6.7 nbconvert-7.0.0 nbformat-5.4.0 nest-asyncio-1.5.5 notebook-6.4.12 notebook-shim-0.1.0 numpy-1.23.2 packaging-21.3 pandas-1.4.4 pandocfilters-1.5.0 parso-0.8.3 pexpect-4.8.0 pickleshare-0.7.5 prometheus-client-0.14.1 prompt-toolkit-3.0.30 psutil-5.9.1 ptyprocess-0.7.0 pure-eval-0.2.2 pycparser-2.21 pygments-2.13.0 pyparsing-3.0.9 pyrsistent-0.18.1 python-dateutil-2.8.2 pytz-2022.2.1 pyzmq-23.2.1 requests-2.28.1 six-1.16.0 sniffio-1.3.0 soupsieve-2.3.2.post1 stack-data-0.5.0 terminado-0.15.0 tinycss2-1.1.1 tornado-6.2 traitlets-5.3.0 urllib3-1.26.12 wcwidth-0.2.5 webencodings-0.5.1 websocket-client-1.4.0 xarray-2022.6.0(testvenv) $
At this point, your venv should be ready.
Using the venv
To launch a jupyterlab session using your venv, you need to fill in a couple of extra fields in the 'advanced' section of the JupyterLab launch form, and also make sure that the job can see the storage your venv is contained in. For example:
Note the highlighted options; you must include the storage option appropriate to the location of your venv, you must specify the same python3 module you loaded when creating the venv, and you must specify the path to the venv.
Troubleshooting
My jupyterlab session doesn't start, and my job card looks like this:
The job either couldn't find your venv, or it didn't have a jupyter command in it, and it also couldn't load the default system jupyterlab module (usually a python conflict)
Check that you have included the right storage options, that the path to your venv is correct, and that your venv does have jupyterlab installed
More details may be visible in the debug log linked to from the job card.
My jupyterlab session starts, but my extra packages aren't available
Check the job card and see where it found the jupyter binary:
If the jupyter binary is showing as being under /apps, the job either couldn't find your venv, or it didn't have a jupyter command in it, but it was able to load the default system jupyterlab module instead.
Check that you have included the right storage options, that the path to your venv is correct, and that your venv does have jupyterlab installed
More details may be visible in the debug log linked to from the job card (once the job is completed; select File -> Shut Down from the JupyterLab menu to cleanly exit the job, and then the debug log link should appear in the job card a minute or two later - it can take a little while after the job completes for the output to be available - try reloading the 'My Interactive Sessions' page if it is taking a long time to appear)
Overview
Content Tools