You can install additional Python packages in a custom directory and make it working together with the NCI-data-analysis module.
Step 1: Load the NCI-data-analysis module in your shell environment
| Code Block | ||
|---|---|---|
| ||
$ module use /g/data/dk92/apps/Modules/modulefiles $ module load NCI-data-analysis/22.05 |
Step 2: Install a python package if it is NOT available in NCI-data-analysis module
There are multiple ways to install Python packages. For example, you could use the pip package manager which is a de facto standard package-management system used to install and manage software packages written in Python (see instruction here: https://packaging.python.org/tutorials/installing-packages/#installing-to-the-user-site ). Or you can build a python library from the source code.
...
| Code Block | ||
|---|---|---|
| ||
[jpf777@gadi-login-09 scr_fp0]$ pip install -v --no-binary :all: --upgrade-strategy only-if-needed --prefix /scratch/fp0/jpf777/EXTRA_PYTHON_LIBS deepgraph Using pip 22.1 from /opt/conda/lib/python3.9/site-packages/pip (python 3.9) Collecting deepgraph Downloading DeepGraph-0.2.3.tar.gz (149 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 149.1/149.1 kB 4.9 MB/s eta 0:00:00 Running command python setup.py egg_info /opt/conda/lib/python3.9/site-packages/setuptools/dist.py:767: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead warnings.warn( running egg_info creating /scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info writing /scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/PKG-INFO writing dependency_links to /scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/dependency_links.txt writing requirements to /scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/requires.txt writing top-level names to /scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/top_level.txt writing manifest file '/scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/SOURCES.txt' reading manifest file '/scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/SOURCES.txt' adding license file 'LICENSE.txt' writing manifest file '/scratch/fp0/jpf777/tmp/pip-pip-egg-info-0fwljpqv/DeepGraph.egg-info/SOURCES.txt' Preparing metadata (setup.py) ... done ... writing requirements to DeepGraph.egg-info/requires.txt writing top-level names to DeepGraph.egg-info/top_level.txt reading manifest file 'DeepGraph.egg-info/SOURCES.txt' adding license file 'LICENSE.txt' writing manifest file 'DeepGraph.egg-info/SOURCES.txt' Copying DeepGraph.egg-info to /scratch/fp0/jpf777/EXTRA_PYTHON_LIBS/lib/python3.9/site-packages/DeepGraph-0.2.3-py3.9.egg-info running install_scripts writing list of installed files to '/scratch/fp0/jpf777/tmp/pip-record-6nwb_evm/install-record.txt' Running setup.py install for deepgraph ... done Successfully installed deepgraph-0.2.3 |
Step 3: Validate new installation
You need to add the custom location to the environment variable PYTHONPATH to use its python libraries together with the NCI-data-analysis module. As show below, the deepgraph comes from the custom location while the dask module still comes from the NCI-data-analysis module itself.
| Code Block | ||
|---|---|---|
| ||
[jpf777@gadi-login-09 scr_fp0]$ export PYTHONPATH=/scratch/fp0/jpf777/EXTRA_PYTHON_LIBS/lib/python3.9/site-packages [jpf777@gadi-login-09 scr_fp0]$ python Python 3.9.10 | packaged by conda-forge | (main, Feb 1 2022, 21:24:11) [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import deepgraph >>> deepgraph.__file__ '/scratch/fp0/jpf777/EXTRA_PYTHON_LIBS/lib/python3.9/site-packages/deepgraph/__init__.py' >>> import dask >>> dask.__file__ '/opt/conda/lib/python3.9/site-packages/dask/__init__.py' >>> exit() |
Step 4: Add PYTHONPATH to your job script
If you want to add the Python packages installed in your own space to your job script, you will need to add in the PYTHONPATH which points to where you installed these packages:
...