Page tree
Skip to end of metadata
Go to start of metadata

Julia is a free, open-sourced high-level, high-performance dynamic programming language for numerical computing. The core of the Julia implementation is licensed under the MIT license. Various libraries used by the Julia environment include their own licenses such as the GPLLGPL, and BSDtherefore, the environment, which consists of the language, user interfaces, and libraries, is under the GPL, see more details on its official site is https://julialang.org

Julia provides a number of key building blocks for distributed computation, making it flexible enough to support a number of styles of parallelism and its just-in-time (JIT) compiler combined with the language’s design allow it to approach and often match the performance of C.  

Julia also provides a built-in package manager Pkg which manages all the packages listed in https://pkg.julialang.org. User can install different packages/modules at different locations named by the environment variable JULIA_PKGDIR for v0.6 and JULIA_DEPOT_PATH for >=v0.7.

Usage

Julia installed on Raijin have been built with Intel MKL to make it run faster. By default it comes with the following packages to support execution on multiple nodes and GPU nodes:

  • MPI
  • CUDAdrv
  • CUDAnative
  • CuArrays


In order to avoid multiple copies of the same package on /short, the module julia/0.6.3 comes with some basic packages, see below an incomplete list

  • Plots
  • GR
  • UnicodePlots
  • StatPlots
  • CSV
  • JSON
  • JLD
  • JLD2
  • MAT
  • NetCDF
  • NCDatasets
  • ImageMagick
  • BenchmarkTools
  • DistributedArrays

Some other popular packages, e.g. DataFrames, might be also installed because of the dependency, to see a full list of site packages, please do 

ls /apps/julia/0.6.3/share/julia/site/v0.6/

From julia/1.1.0, please see the list in the file $JULIA_ROOT/share/julia/site/environments/site/Project.toml.

We expect that other packages will be installed by users in their own directory under either /home or /short, see the section below for an example of how to install the package of NLopt.


To load the specific version of 1.1.0 please do

module load julia/1.1.0

An example job submission script (juliajob.sh) is provided below. It requests 32 cores, 64GB of memory, 200GB of local disk space, and 1 hours of walltime for the job. This submission script should be saved in the working directory from which the analysis will be done. To change the number of processors, memory, or jobfs required, simply modify the appropriate PBS resource requests at the top of this file.

#!/bin/bash
#PBS -q normal
#PBS -l ncpus=32
#PBS -l mem=64GB
#PBS -l jobfs=200GB
#PBS -l walltime=1:00:00
#PBS -l software=julia
#PBS -l wd

# Load modules.
module load julia/1.1.0

# Run julia applications
julia -np $PBS_NCPUS cputest.jl > $PBS_JOBID.log

Please note, the built-in Cluster Managers doesn't work on Raijin because Julia by default uses 'ssh' to establish/kill multiple processes across multiple nodes and manage the communications among them, but Raijin doesn't support ssh between any compute nodes. Instead, we utilise the MPIManager in the MPI.jl package to allow julia applications running on multiple compute nodes on Raijin, please refer to the following section for how to run julia on multiple cores.   


Once the job submission script is ready, please do 

qsub juliajob.sh 

Running Julia on multiple nodes

An example julia script cputest.jl is provided below. It starts 32 processes to summarise the 10^3 random Boolean matrices generated using all the 32 cores. 

using MPI, Distributed

MPI.Init()
manager = MPI.start_main_loop(MPI_TRANSPORT_ALL)

fn = "prand.log"
out = @distributed (append!) for i in 1:10^3 
  pid=getpid()
  ps = readlines(pipeline(`ps -opid,psr`,`grep $pid`))
  pid, core = split(sp[1]) 
  [join(["generating ", rand([-1,0,1])," on (pid,core)=($pid,$core) on node ", gethostname(),"\n"]),] 
end
write(fn,join(out))
MPI.stop_main_loop(manager)

Installing Julia packages

To install a package maintained in the Julia Package Listing, please see an example below. It installs the nonlinear optimisation package NLopt.jl into your project folder under /short using the native Pkg module from the Julia standard library and then tests the installation using the built-in test files in the new package. 

login node $ module load julia/1.1.0
login node $ module load gcc/6.2.0
login node $ mkdir /short/$PROJECT/$USER/.julia
login node $ export JULIA_DEPOT_PATH=/short/$PROJECT/$USER/.julia:$JULIA_ROOT/share/julia/site/
login node $ export JULIA_LOAD_PATH="@":"@v#.#":"@stdlib":"@site"
login node $ julia
julia > ]
(v1.1) pkg> add NLopt

Please also note for packages need to be built on Raijin, do module load the dependencies inside your environment, such as module load gcc/6.2.0, and set the environment variables required before calling julia. 

To automatically update the environment variable JULIA_DEPOT_PATH and JULIA_LOAD_PATH every time you login to Raijin, please set

export JULIA_DEPOT_PATH=/short/$PROJECT/$USER/.julia:$JULIA_ROOT/share/julia/site/
export JULIA_LOAD_PATH="@":"@v#.#":"@stdlib":"@site"
(or setenv if you are using tcsh)

in .profile (or .login for tcsh users) file in your home directory.