Page tree
Skip to end of metadata
Go to start of metadata

You might need to customise your environment for reasons including:

  • The Python libraries you require are not included in the current Pangeo and/or NCI-data-analysis virtual environments available in dk92
  • You develop your own code

Users can install additional Python packages under their own working directory, or their computer project space if it is permitted. Please note, the Pangeo environment should always be loaded before adding other modules or installing new packages.

Step 1: Enable Pangeo in your shell environment

To enable the Pangeo environment, you can use the following command within jobs, or within an interactive environment:

$ module load pangeo/2021.01
Loading pangeo/2021.01
  Loading requirement: intel-mkl/2019.3.199 python3/3.7.4 hdf5/1.10.5
    netcdf/4.7.3

Step 2: Install modules if they are NOT available in dk92

There are multiple ways to install Python packages. For example, you could use the pip package manager which is a de facto standard package-management system used to install and manage software packages written in Python (see instruction here: https://packaging.python.org/tutorials/installing-packages/#installing-to-the-user-site ). Another popular way to install packages is through Conda which is an open source package, dependency and environment management system for any language - Python, R, Ruby, Jua, Scala, Java, JavaScript, C/C++, FORTRAN and many more. See some examples for a quick start.

Please note: additional packages should be installed within a user’s own directory

Let’s now install the Deep Graph Library using pip:

[abc123@gadi-login-01 ~]$ pip install --user deepgraph

Collecting deepgraph
  Using cached https://files.pythonhosted.org/packages/fc/3e/4a34a5316a5f886b8d7a6787c24852d9e5a5ef00b4ec6af0736f681a3a58/DeepGraph-0.2.2.tar.gz
Requirement already satisfied: numpy>=1.6 in /apps/python3/3.7.4/lib/python3.7/site-packages/numpy-1.17.2-py3.7-linux-x86_64.egg (from deepgraph) (1.17.2)
Requirement already satisfied: pandas>=0.17.0 in /apps/pangeo/2019.12/lib/python3.7/site-packages (from deepgraph) (0.25.3)
Requirement already satisfied: pytz>=2017.2 in /apps/pangeo/2019.12/lib/python3.7/site-packages (from pandas>=0.17.0->deepgraph) (2019.3)
Requirement already satisfied: python-dateutil>=2.6.1 in /apps/python3/3.7.4/lib/python3.7/site-packages (from pandas>=0.17.0->deepgraph) (2.8.1)
Requirement already satisfied: six>=1.5 in /apps/python3/3.7.4/lib/python3.7/site-packages (from python-dateutil>=2.6.1->pandas>=0.17.0->deepgraph) (1.13.0)
Building wheels for collected packages: deepgraph
  Building wheel for deepgraph (setup.py) ... done
  Created wheel for deepgraph: filename=DeepGraph-0.2.2-cp37-cp37m-linux_x86_64.whl size=373893 sha256=53e6966cdd833e99af226dd925f9d9f1a10259053cd13f4391caa356bbfedabb
  Stored in directory: /home/900/nre900/.cache/pip/wheels/7f/4b/45/caf95420067f7a1795c5664bce0beda747d0ce931c2424c5ff
Successfully built deepgraph
Installing collected packages: deepgraph
Successfully installed deepgraph-0.2.2

Step 3: Validate new installation

[abc123@gadi-login-01 ~]$ python 
Python 3.7.4 (default, Nov 6 2019, 12:34:08) [GCC 8.2.1 20180905 (Red Hat 8.2.1-3)] on linux Type "help", "copyright", "credits" or "license" for more information. 
>>> import deepgraph 
>>> exit()


Step 4: Add PYTHONPATH to your job script

If you want to add the Python packages installed in your own space to your job script, you will need to add in the PYTHONPATH which points to where you installed these packages:

#!/bin/bash
#PBS -N pangeo_test
#PBS -P <project code>
#PBS -q normal
#PBS -l walltime=5:00:00
#PBS -l ncpus=96
#PBS -l mem=384GB
#PBS -l jobfs=100GB
#PBS -l storage=scratch/z00+scratch/<project code>+gdata/<project code>
#PBS -v PYTHONPATH=<path to where you installed your Python packages> 

module use /g/data/dk92/apps/Modules/modulefiles
module load pangeo/2021.01
module load NCI-data-analysis
pangeo.ini.all.sh
sleep infinity



  • No labels