Specialised Environments

Page tree


Cromwell is a workflow management system that can be used to run WDL workflows. Support for cromwell at NCI is in its early stages and may change in the future.

Resources available at NCI

1. cromwell module

As part of project hr32, a module is available that includes:

  • cromwell, a java application (distributed as a jar file) for starting a cromwell server or running a workflow directly.
  • womtool, a java application for checking workflows and creating input files.
  • cromshell, a popular command line tool for interacting with a cromwell server.
  • An example configuration file (cromwell.conf) that can be used to set up a standalone cromwell server or run workflows on gadi using PBS.

This module can be loaded with the command:

module use /g/data/hr32/apps/Modules/modulefiles        # First add hr32 to the module path
module load cromwell/69

Cromwell configuration file

To support the PBS queuing system, a custom configuration file has been created that can be further customised for use with the PBS queue system on gadi. Once the cromwell module is loaded, the file is at ${CROMWELL_HOME}/share/cromwell.conf. A similar file is used to configure cromwell within the cromwell OOD app. 

For resource specification, the following variables are available:

variable namedefault valuecommentcorresponding PBS string
cpu1Number of cpus-lncpus=${cpu}
mem1Required memory in GB-lmem=${mem}G
queue-Optional, specifies the queue to submit to.-q ${queue}
project-Job project-P ${project}
walltime"10:00:00"Job walltime-lwalltime=${walltime}
umask"umask=0007"The umask attribute for job standard output and error-W ${umask}
clusterOptions"wd"Can be used to set storage flags or request GPUs-l${clusterOptions}

An example runtime section within a task definition might be:

  runtime {
    queue: "normal"
    project: "ab123"
    cpu: 2
    walltime: "24:00:00"
    mem: 10
    clusterOptions: "storage=gdata/ab123"
  }

Using the configuration file

Several variables need to be passed to cromwell in order to use the configuration file. These set directories for logging and execution. Demonstrations of how to set these are in the following examples.

# Run the cromwell server on port 8000
module use /g/data/hr32/apps/Modules/modulefiles
module load cromwell
module load java/jdk-13.33
CROMWELL_JAR=${CROMWELL_HOME}/cromwell.jar
CROMWELL_EXECUTION=/scratch/ab123/cromwell-execution
CROMWELL_LOG_DIR=/scratch/ab123/cromwell-logs

java -Dwebservice.port=8000 \
     -Dconfig.file=${CROMWELL_HOME}/share/cromwell.conf \
     -Dbackend.providers.TORQUE.config.root=${CROMWELL_EXECUTION} \
     -Dworkflow-options.workflow-log-dir=${CROMWELL_LOG_DIR} \
     -jar ${CROMWELL_JAR} server
# Run a workflow on cromwell directly
module use /g/data/hr32/apps/Modules/modulefiles
module load cromwell
module load java/jdk-13.33
CROMWELL_JAR=${CROMWELL_HOME}/cromwell.jar
CROMWELL_EXECUTION=/scratch/ab123/cromwell-execution
CROMWELL_LOG_DIR=/scratch/ab123/cromwell-logs

java -Dconfig.file=${CROMWELL_HOME}/share/cromwell.conf \
     -Dbackend.providers.TORQUE.config.root=${CROMWELL_EXECUTION} \
     -Dworkflow-options.workflow-log-dir=${CROMWELL_LOG_DIR} \
     -jar ${CROMWELL_JAR} run <workflow-arguments>


  • No labels