Software Applications on /apps
Gadi has a large array of software applications installed on its system. These are found in /apps and uses environment modules to manage them. This page will help you to learn the basics of navigating /apps and loading modules for use on Gadi.
Searching for Applications
Once you are logged into Gadi, a good first is to simply see the array of what modules are available in /apps. you can do this by running the command
$ ls -l /apps/
As you can see, this has produced a very long list of every module that is available on Gadi, 940 at this point in time. You can scroll this list and find the right module for you or you can go into more detail for specific apps by running the command
$ ls -l /apps/<appname>
replacing <appname> with the module you are investigating.
This produces a list of the different versions of that particular module that are installed on gadi.
You can take this a step further and get a listing of every version of each module by running the command
$ module avail
This listing is extensive but can be useful if your are browsing for a particular version of an module.
You can also narrow these results and search this listing. If you look at the image on the right you can see that /intel has been entered in the bottom left of this listing. This will search for phrase you have entered and highlight the results for you.
If you find a module in this list and want to focus in on it, simply run the command
$ module avail <appname>
For instance, running
$ module avail python3
will only return the different versions of python3 that are installed on Gadi. When possible, NCI recommends loading a specific version of a module that you know is compatible with your code.
Loading Modules
Let's say that you wanted to load python3, there are two ways to do this. You can run the command
$ module load python3
This will automatically load the latest stable version of python, or what is considered to be the most stable version, in this case, it's python3/3.11.0. However, as mentioned above, loading a specific version of an app that you know is compatible with your code is recommended. You can do this by adding the version number after the module name, for example,
module load python3/3.11.0
This limits the chance of your job failing because of a software incompatibility. For example, you may have run your binary a month ago, but NCI has updated the software since then and it no longer runs on the default. Loading specific versions reduces the chances of job failure on Gadi.
You'll notice that it has also loaded intel-mkl/2022.2.0, this is called a dependency. For python3 to operate properly, it needs access to that library. These dependencies will load automatically for you.
If you want to see what modules you have loaded you can use
$ module list
As you can see on the right, python3 and intel-mkl are currently loaded, but so is pbs. pbs is automatically loaded when logging into Gadi, as it assumes you have logged in to run pbs jobs.
Gadi won't allow you to run more than one version of a module at a time. The image on the right shows what happens when a different version of python3 is requested while python3 is already running.
This error comes with a hint at the bottom too, 'Might try module unload python3' first.
To do this you run the command
$ module unload python3
This will remove the first version that you loaded and allow you to load the new one.
Note: The depencies that were loaded with the first version will remain.
If you would like to see the breakdown of a module, including all its paths and locations, you can run
$ module show <appname>
as you can see on the right, this gives and in-depth look into the app.
Module Commands
Command | Notes |
|---|---|
| module avail <str> | List all available modulefiles in the current MODULEPATH whose pathname starts with <str>. All directories in the MODULEPATH are recursively searched for the target files that contain the modulefile magic cookie. If no argument <str> is given, all the modulefiles are displayed. |
| module list | List all loaded modules in the current shell environment |
| module load <app>/<version> | Load modulefile <app>/<version> into the current shell environment. |
| module purge | Remove all modulefiles from the current shell environment. |
| module unload <app> | Remove modulefile for application <app> in the current shell |
| module whatis <app>/<version> | Display the information set up by the module-whatis commands inside the modulefile <app>/<version>. |
User Defined Modules
Users can edit their own modulefiles for different versions of applications in their home folder ~/privatemodules. Once the modulefile is created in the `privatemodules` folder, first run
$ module load use.own
to include `~/privatemodules` in the search path MODULEPATH and prepare the reference counter to start tracking it, then run `module load <modulefile>` as you would load other modulefiles for applications installed on /apps.
For example, after installing python packages `neural-structured-learning`, with the module `python3/3.7.4` and `tensorflow/2.0.0` loaded, to a site-packages folder in your home folder, such as
$ module purge $ unset PYTHONPATH $ module load python3/3.7.4 $ module load tensorflow/2.0.0 $ pip3 install -v --no-binary :all: --prefix=$HOME/envs/nsl/1.1.0 neural-structured-learning==1.1.0
#%Module1.0 prereq python3/3.7.4 prereq tensorflow/2.0.0 prepend-path PYTHONPATH [getenv HOME]/envs/nsl/1.1.0/lib/python3.7/site-packages
so that loading the modulefile `nsl/1.1.0` gets the environment prepared for jobs need neural-structured-learning version 1.1.0 imported. For example,
$ module load use.own $ module load python3/3.7.4 $ module load tensorflow/2.0.0 $ module load nsl/1.1.0 $ python3 >> import tensorflow as tf >> import neural_structured_learning as nsl >> nsl.__file__ '~/envs/nsl/1.1.0/lib/python3.7/site-packages/neural_structured_learning/__init__.py'
Authors: Yue Sun, Mohsin Ali, Andrew Johnston
Overview
Content Tools