You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 10 Next »

You can install additional Python packages in a custom directory and make it working together with the NCI-data-analysis module. 

Step 1: Load the NCI-data-analysis module in your shell environment

$ module use /g/data/dk92/apps/Modules/modulefiles
$ module load NCI-data-analysis/2022.06

Step 2: Install a python package if it is NOT available in NCI-data-analysis module

There are multiple ways to install Python packages. For example, you could use the pip package manager which is a de facto standard package-management system used to install and manage software packages written in Python (see instruction here: https://packaging.python.org/tutorials/installing-packages/#installing-to-the-user-site ). Or you can build a python library from the source code.  

Please note: additional packages should be installed within a user’s own directory

Let’s now install the Deep Graph Library using pip.


First of all, make sure the Deep Graph Library is not available in the NCI-data-analysis module,

[jpf777@gadi-login-09 scr_fp0]$ python
Python 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:24:11)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import deepgraph
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'deepgraph'
>>> exit()

Then we can use 'pip' command to install the Deep Graph Library into a custom directory ( i.e. /scratch/fp0/jpf777/EXTRA_PYTHON_LIBS as below). We recommend to try the following flags to build the library. To learn more flags of pip command, please visit here.

[jpf777@gadi-login-09 scr_fp0]$ pip install -v --no-binary :all: --upgrade-strategy only-if-needed --prefix /scratch/fp0/jpf777/EXTRA_PYTHON_LIBS deepgraph
Using pip 22.1 from /opt/conda/lib/python3.9/site-packages/pip (python 3.9)
Collecting deepgraph
Downloading DeepGraph-0.2.3.tar.gz (149 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 149.1/149.1 kB 4.9 MB/s eta 0:00:00
Running command python setup.py egg_info
/opt/conda/lib/python3.9/site-packages/setuptools/dist.py:767: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
warnings.warn(
running egg_info
creating /scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info
writing /scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/PKG-INFO
writing dependency_links to /scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/dependency_links.txt
writing requirements to /scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/requires.txt
writing top-level names to /scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/top_level.txt
writing manifest file '/scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/SOURCES.txt'
reading manifest file '/scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/SOURCES.txt'
adding license file 'LICENSE.txt'
writing manifest file '/scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/SOURCES.txt'
Preparing metadata (setup.py) ... done
...
writing requirements to DeepGraph.egg-info/requires.txt
writing top-level names to DeepGraph.egg-info/top_level.txt
reading manifest file 'DeepGraph.egg-info/SOURCES.txt'
adding license file 'LICENSE.txt'
writing manifest file 'DeepGraph.egg-info/SOURCES.txt'
Copying DeepGraph.egg-info to /scratch/fp0/jpf777/EXTRA_PYTHON_LIBS/lib/python3.9/site-packages/DeepGraph-0.2.3-py3.9.egg-info
running install_scripts
writing list of installed files to '/scratch/fp0/jpf777/tmp/pip-record-6nwb_evm/install-record.txt'
Running setup.py install for deepgraph ... done
Successfully installed deepgraph-0.2.3

Step 3: Validate new installation

You need to add the custom location to the environment variable PYTHONPATH to use its python libraries together with the NCI-data-analysis module. As show below, the deepgraph comes from the custom location while the dask module still comes from the NCI-data-analysis module itself.

[jpf777@gadi-login-09 scr_fp0]$ export PYTHONPATH=/scratch/fp0/jpf777/EXTRA_PYTHON_LIBS/lib/python3.9/site-packages
[jpf777@gadi-login-09 scr_fp0]$ python
Python 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:24:11)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import deepgraph
>>> deepgraph.__file__
'/scratch/fp0/jpf777/EXTRA_PYTHON_LIBS/lib/python3.9/site-packages/deepgraph/__init__.py'
>>> import dask
>>> dask.__file__
'/opt/conda/lib/python3.9/site-packages/dask/__init__.py'
>>> exit()


Step 4: Add PYTHONPATH to your job script

If you want to add the Python packages installed in your own space to your job script, you will need to add in the PYTHONPATH which points to where you installed these packages:

#!/bin/bash
#PBS -N python-test
#PBS -P <project code>
#PBS -q normal
#PBS -l walltime=5:00:00
#PBS -l ncpus=96
#PBS -l mem=384GB
#PBS -l jobfs=100GB
#PBS -l storage=gdata/dk92+scratch/<project code>+gdata/<project code>
#PBS -v PYTHONPATH=<path to where you installed your Python packages> 

module use /g/data/dk92/apps/Modules/modulefiles
module load NCI-data-analysis/22.05
RUN_YOUR_OWN_PYTHON_SCRIPT



  • No labels