User can use their own Julia packages if they need more than the default ones provided by the `NCI-data-analysis` module under the project dk92, no matter if the julia session is running from an OOD VDI Desktop, within the ARE jupyter notebook or through command line on a gadi node.
Please see examples below for package installation into your project folder under /g/data.
OOD VDI
On the start of a Julia session, it uses packages provided by the module. The default location `/opt/julia` exists only in the singularity image and no modification is allowed. New packages have to be installed to another location with write permission.
To use another environment, add the path as the first element in `DEPOT_PATH`. In our example below, we choose to build the new environment in `/g/data/z00/.julia`.
When it is a new environment, the initialisation is required. An empty `Project.toml` file in the directory `/g/data/z00/.julia/environments/v<version>/` is the minimum requirement for the Julia package manager to find the new environment. Once found, it is ready to add the missing package to the new location.
To shift back to the default environment, simply pop the first element out of `DEPOT_PATH`.
[abc123@ood-vn1 ~]$ module use /g/data/dk92/apps/Modules/modulefiles
[abc123@ood-vn1 ~]$ module load NCI-data-analysis/2022.06
[abc123@ood-vn1 ~]$ julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.7.2 (2022-02-06)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> DEPOT_PATH
1-element Vector{String}:
"/opt/julia"
(@v1.7) pkg> st GraphIO
No Matches in `/opt/julia/environments/v1.7/Project.toml`
julia> ENV["PROJECT"]
"z00"
julia> pushfirst!(DEPOT_PATH,"/g/data/"*ENV["PROJECT"]*"/.julia")
2-element Vector{String}:
"/g/data/z00/.julia"
"/opt/julia"
shell> mkdir -p /g/data/z00/.julia/environments/v1.7
shell> touch /g/data/z00/.julia/environments/v1.7/Project.toml
(v1.7) pkg> st
Status `/g/data4/z00/.julia/environments/v1.7/Project.toml` (empty project)
Resolving package versions...
Installed Compat ------------ v4.1.0
Installed ArnoldiMethod ----- v0.2.0
Installed GraphIO ----------- v0.6.0
Installed Graphs ------------ v1.5.0
Updating `/g/data4/z00/.julia/environments/v1.7/Project.toml`
[aa1b3936] + GraphIO v0.6.0
Updating `/g/data4/z00/.julia/environments/v1.7/Manifest.toml`
...
Precompiling project...
5 dependencies successfully precompiled in 16 seconds (9 already precompiled)
(v1.7) pkg> test GraphIO
Testing GraphIO
Status `/local/z00/abc123/tmp/jl_jswoue/Project.toml`
[944b1d66] CodecZlib v0.7.0
[8f5d6c58] EzXML v1.1.0
[aa1b3936] GraphIO v0.6.0
[86223c79] Graphs v1.5.0
[fae87a5f] ParserCombinator v2.1.1
[ae029012] Requires v1.3.0
[699a6c99] SimpleTraits v0.9.4
[8bb1440f] DelimitedFiles `@stdlib/DelimitedFiles`
[8dfed614] Test `@stdlib/Test`
Status `/local/z00/abc123/tmp/jl_jswoue/Manifest.toml`
[ec485272] ArnoldiMethod v0.2.0
...
Test Summary: | Pass Total
GraphIO | 152 152
Testing GraphIO tests pass
julia> popfirst!(DEPOT_PATH)
"/g/data/z00/.julia"
(@v1.7) pkg> st Pluto
Status `/opt/julia/environments/v1.7/Project.toml`
[c3e4b0f8] Pluto v0.14.7
ARE Jupyter Notebooks
To install a package maintained in the Julia Package Listing requires internet access. Unlike Julia running in an OOD session which by default has the access to the external network, Julia notebook has to be running inside an ARE session that requests resources from `copyq` queue to get the internet access.
The above example is also available in this notebook.
Gadi Nodes
Not all gadi nodes have the access to the external network. Ideally, it is feasible to install and test Julia packages on the login node. However, If the installation or the test is too big to run on a login node and doesn't comply with the login node resource usage limit, simply submit an interactive copyq job and do the installation/test from there.
The environment variable `SINGULARITYENV_JULIA_DEPOT_PATH` passes in its value to the environment variable `JULIA_DEPOT_PATH` within the singularity image in which julia and all its packages packages are installed. It overwrites the default value `/opt/julia`. If at any point a package installed in the module is required, simply point back to that Project.toml file by inserting the path back to the `DEPOT_PATH` vector as the first element.
login node $ JULIA_PKGDIR=/g/data/$PROJECT/$USER/.julia.$(date +%Y).$(date +%m).$(date +%d).$(date +%H)
login node $ mkdir -p $JULIA_PKGDIR
login node $ export SINGULARITYENV_JULIA_DEPOT_PATH=$JULIA_PKGDIR
login node $ module use /g/data/dk92/apps/Modules/modulefiles
login node $ module load NCI-data-analysis/22.06
login node $ julia
julia > DEPOT_PATH
1-element Vector{String}:
"/g/data/z00/abc123/.julia.2022.06.21.10"
(@v1.7) pkg> st
Installing known registries into `/g/data/z00/abc123/.julia.2022.06.21.10`
Status `/g/data/z00/abc123/.julia.2022.06.21.10/environments/v1.7/Project.toml` (empty project)
(@v1.7) pkg> add GraphIO
Updating registry at `/g/data/z00/abc123/.julia.2022.06.21.10/registries/General.toml`
Resolving package versions...
...
Updating `/g/data/z00/abc123/.julia.2022.06.21.10/environments/v1.7/Project.toml`
[aa1b3936] + GraphIO v0.6.0
Updating `/g/data/z00/abc123/.julia.2022.06.21.10/environments/v1.7/Manifest.toml`
...
Precompiling project...
14 dependencies successfully precompiled in 11 seconds
(@v1.7) pkg> test GraphIO
Testing GraphIO
Status `/scratch/z00/abc123/tmp/jl_MbtVIQ/Project.toml`
...
Precompiling project...
12 dependencies successfully precompiled in 3 seconds (14 already precompiled)
Testing Running tests...
...
Test Summary: | Pass Total
GraphIO | 152 152
Testing GraphIO tests passed
julia> using GraphIO
julia> names(GraphIO)
9-element Vector{Symbol}:
:CDFFormat
:DOTFormat
:EdgeListFormat
:GEXFFormat
:GMLFormat
:Graph6Format
:GraphIO
:GraphMLFormat
:NETFormat
julia> pushfirst!(DEPOT_PATH,"/opt/julia")
2-element Vector{String}:
"/opt/julia"
"/g/data/z00/yxs900/.julia.2022.06.21.10"
(@v1.7) pkg> st Graphs
Status `/opt/julia/environments/v1.7/Project.toml`
[86223c79] Graphs v1.7.0
julia> using Graphs