Specialised Environments

Page tree

You can install additional Python packages in a custom directory and make it working together with the NCI-data-analysis module. 

Step 1: Load the NCI-data-analysis module in your shell environment

$ module use /g/data/dk92/apps/Modules/modulefiles
$ module load NCI-data-analysis/2022.06

Step 2: Install a python package if it is NOT available in NCI-data-analysis module

There are multiple ways to install Python packages. For example, you could use the pip package manager which is a de facto standard package-management system used to install and manage software packages written in Python (see instruction here: https://packaging.python.org/tutorials/installing-packages/#installing-to-the-user-site ). Or you can build a python library from the source code.  

Please note: additional packages should be installed within a user’s own directory.

Let’s now install the Deep Graph Library using pip.


First of all, make sure the Deep Graph Library is not available in the NCI-data-analysis module,

[jpf777@gadi-login-09 scr_fp0]$ python
Python 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:24:11)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import deepgraph
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'deepgraph'
>>> exit()

Then we can use 'pip' command to install the Deep Graph Library into a custom directory ( i.e. /scratch/fp0/jpf777/EXTRA_PYTHON_LIBS as below). We recommend to try the following flags to build the library. To learn more flags of pip command, please visit here.

[jpf777@gadi-login-09 scr_fp0]$ pip install -v --no-binary :all: --upgrade-strategy only-if-needed --prefix /scratch/fp0/jpf777/EXTRA_PYTHON_LIBS deepgraph
Using pip 22.1 from /opt/conda/lib/python3.9/site-packages/pip (python 3.9)
Collecting deepgraph
Downloading DeepGraph-0.2.3.tar.gz (149 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 149.1/149.1 kB 4.9 MB/s eta 0:00:00
Running command python setup.py egg_info
/opt/conda/lib/python3.9/site-packages/setuptools/dist.py:767: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
warnings.warn(
running egg_info
creating /scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info
writing /scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/PKG-INFO
writing dependency_links to /scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/dependency_links.txt
writing requirements to /scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/requires.txt
writing top-level names to /scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/top_level.txt
writing manifest file '/scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/SOURCES.txt'
reading manifest file '/scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/SOURCES.txt'
adding license file 'LICENSE.txt'
writing manifest file '/scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/SOURCES.txt'
Preparing metadata (setup.py) ... done
...
writing requirements to DeepGraph.egg-info/requires.txt
writing top-level names to DeepGraph.egg-info/top_level.txt
reading manifest file 'DeepGraph.egg-info/SOURCES.txt'
adding license file 'LICENSE.txt'
writing manifest file 'DeepGraph.egg-info/SOURCES.txt'
Copying DeepGraph.egg-info to /scratch/fp0/jpf777/EXTRA_PYTHON_LIBS/lib/python3.9/site-packages/DeepGraph-0.2.3-py3.9.egg-info
running install_scripts
writing list of installed files to '/scratch/fp0/jpf777/tmp/pip-record-6nwb_evm/install-record.txt'
Running setup.py install for deepgraph ... done
Successfully installed deepgraph-0.2.3

Step 3: Validate new installation

You need to add the custom location to the environment variable PYTHONPATH to use its python libraries together with the NCI-data-analysis module. As show below, the deepgraph comes from the custom location while the dask module still comes from the NCI-data-analysis module itself.

[jpf777@gadi-login-09 scr_fp0]$ export PYTHONPATH=/scratch/fp0/jpf777/EXTRA_PYTHON_LIBS/lib/python3.9/site-packages
[jpf777@gadi-login-09 scr_fp0]$ python
Python 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:24:11)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import deepgraph
>>> deepgraph.__file__
'/scratch/fp0/jpf777/EXTRA_PYTHON_LIBS/lib/python3.9/site-packages/deepgraph/__init__.py'
>>> import dask
>>> dask.__file__
'/opt/conda/lib/python3.9/site-packages/dask/__init__.py'
>>> exit()


Step 4: Setting PYTHONPATH to a PBS job script

You can add your own Python package location to the environment variable PYTHONPATH in a PBS job script as below

#!/bin/bash
#PBS -N python-test
#PBS -P <project code>
#PBS -q normal
#PBS -l walltime=5:00:00
#PBS -l ncpus=96
#PBS -l mem=384GB
#PBS -l jobfs=100GB
#PBS -l storage=gdata/dk92+scratch/<project code>+gdata/<project code>
#PBS -v PYTHONPATH=<path to where you installed your Python packages> 

module use /g/data/dk92/apps/Modules/modulefiles
module load NCI-data-analysis/2022.06
RUN_YOUR_OWN_PYTHON_SCRIPT



  • No labels