Date: Tue, 19 Mar 2024 13:07:07 +1100 (AEDT) Message-ID: <1406352503.967.1710814027785@opus-prod1.nci.org.au> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_966_1704784018.1710814027784" ------=_Part_966_1704784018.1710814027784 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html
Gadi uses Environment Modules to customise user shell environment. The Environment Modules p= ackage is a tool that simplifies shell initialisation and allow users to mo= dify their environments using provided commands like `module load` and `mod= ule unload` during the session with modulefiles.
On Gadi, we prepare
Typically modulefiles instru= ct the module command to alter or set shell environment variables such as P= ATH. To look up the instructions in the modulefile, run `module show`. = ;For example, the= modulefile for application version python3/3.7.4 contains the following in= structions.
$ modul= e show python3/3.7.4 ------------------------------------------------------------------- /apps/Modules/modulefiles/python3/3.7.4: prepend-path PATH /apps/python3/3.7.4/bin prepend-path C_INCLUDE_PATH /apps/python3/3.7.4/include/python3.7m prepend-path CPLUS_INCLUDE_PATH /apps/python3/3.7.4/include/python3.7m prepend-path CPATH /apps/python3/3.7.4/include/python3.7m prepend-path FPATH /apps/python3/3.7.4/include/python3.7m prepend-path LIBRARY_PATH /apps/python3/3.7.4/lib prepend-path LD_LIBRARY_PATH /apps/python3/3.7.4/lib prepend-path LD_RUN_PATH /apps/python3/3.7.4/lib prepend-path MANPATH /apps/python3/3.7.4/share/man prepend-path PKG_CONFIG_PATH /apps/python3/3.7.4/lib/pkgconfig module load intel-mkl/2019.3.199 conflict python3 setenv PYTHON3_BASE /apps/python3/3.7.4 setenv PYTHON3_ROOT /apps/python3/3.7.4 setenv PYTHON3_VERSION 3.7.4 module-whatis {python3, version 3.7.4} -------------------------------------------------------------------
It tells the command `module= load` to set the environment variables PYTHON3_BASE, PYTHON3_ROOT, and PYT= HON3_VERSION to the corresponding values.
It also tells `module load` = to prepend the `bin` folder to the environment variable PATH in order to le= t the shell environment know where to find the executables provided by the = python3/3.7.4 installation. There are also instructions for adding the path= to the folder that contains the header files, `/apps/python3/3.7.4/include/python3.7m`, to = the environment variables C_INCLUDE_PATH, CPLUS_INCLU= DE_PATH, CPATH, and FPATH, and the path to the folder that c= ontains libraries, `/apps/python3/3.7.4/lib`, to LIBRARY_PATH, LD= _LIBRARY_PATH, and LD_RUN_PATH in order to let the shell environm= ent know where to find the header files and libraries, respectively, that a= re expected to be part of python version 3.7.4. There are also pat= hs added to MANPATH and PKG_CONFIG_PATH to tell the command `man` and `pkg-= config` where to search for python3/3.7.4 related man pages and .pc files, = respectively.
In the modulefile, it also says `module load intel-mkl/2019.3.199`. This= instruction tells when loading the module python3/3.7.4, load the module i= ntel-mkl/2019.3.199 together with it. We recommend this configuration= because python3/3.7.4 is built with intel-mkl libraries and it is necessar= y to set environment variables for it too.
Since there are going to be multiple versions of the same application in= stalled on Gadi and, in a certain shell, only one can be used, it is import= ant to tell that all other module versions conflict with the current one. I= n the above example, `conflict python3` instructs the `module load` command= not to load any other python3 modules if the module python3/3.7.4 has been= already loaded to the current shell.
Users are encouraged to load modulefiles with their fullnames, <app&g= t;/<version>, to remove ambiguity and improve the reproducibility of = works on Gadi, especially in job submission scripts and their private modul= efiles. Because most of software applications Gadi keeps centrally on= /apps are expected to have multiple versions available and we keep install= ing newly released versions, it is the best to load modulefiles with the sp= ecific version to avoid using a wrong version when resuming/replicating pre= vious works on Gadi. For example,
$ modul= e load openmpi $ module list Currently Loaded Modulefiles: 1) openmpi/3.1.4 ... ################### # 3 months later # ################### ... $ module load openmpi $ module list Currently Loaded Modulefiles: 1) openmpi/4.0.2
We always set the default version of the application to the most recent = or the most reliable version, therefore, it is expected that `module load o= penmpi` loads different versions of the openmpi application overtime. If yo= ur binary is compiled with openmpi/3.1.4 and the job submission script has = `module load openmpi` in it, it may run at the time version 3.1.4 is still = the default version, but it could fail 3 months later when the newer versio= n is installed and set to the default version for openmpi applications.
Command | Notes |
---|---|
module list | List all load= ed modules in the current shell environment |
module purge | Remove all mo= dulefiles from the current shell environment. |
module load <app>/<version> | Load modulefi= le <app>/<version> into the current shell environment.= td> |
module unload <app> | Remove module= file for application <app> in the current shell |
module avail <str> = &nbs= p; | List all avai= lable modulefiles in the current MODULEPATH whose pathname starts with <= str>. All directories in the MODULEPATH are recursively searched for the= target files that contain the modulefile magic cookie. If no argument= <str> is given, all the modulefiles are displayed. |
module whatis <app>/<version> | Display the i= nformation set up by the module-whatis commands inside the modulefile <a= pp>/<version>. |
Users can edit their own&nbs= p;modulefiles for different versions of applications in their home folder ~= /privatemodules. Once the modulefile <modulefile> is created in the `= privatemodules` folder, first run
$ modul= e load use.own
to include `~/privatemodules`= in the search path MODULEPATH and prepare the reference counter to start t= racking it, then run `module load <modulefile>` as y= ou would load other modulefiles for applications installed on /apps<= /span>.
For example, after installing python packages `neural-structured-learning`= , with the module `python3/3.7.4` and `tensorflow/2.0.0` loaded, to a= site-packages folder in your home folder, like
$ modul= e purge $ unset PYTHONPATH $ module load python3/3.7.4 $ module load tensorflow/2.0.0 $ pip3 install -v --no-binary :all: --prefix=3D$HOME/envs/nsl/1.1.0 neural-= structured-learning=3D=3D1.1.0
edit the modulefile ~/privatemodules/nsl/1.1.0 as
#%Modul= e1.0 prereq python3/3.7.4 prereq tensorflow/2.0.0 prepend-path PYTHONPATH [getenv HOME]/envs/nsl/1.1.0/lib/python3.7/site-= packages
so that loading the modulefile `nsl/1.1.0= ` gets the environment prepared for jobs need neural-structured-learning ve= rsion 1.1.0 imported. For example,
$ modul= e load use.own $ module load python3/3.7.4 $ module load tensorflow/2.0.0 $ module load nsl/1.1.0 $ python3 >> import tensorflow as tf >> import neural_structured_learning as nsl >> nsl.__file__ '~/envs/nsl/1.1.0/lib/python3.7/site-packages/neural_structured_learning/__= init__.py'