Page History

Overview

VASP stands for Vienna Ab-initio Simulation Package, and is a package for performing ab-initio quantum-mechanical molecular dynamics (MD) using pseudopotentials and a plane wave basis set.

VASP is distributed though Prof. Juergen Hafner. See the VASP web site for more details.

Usage

Once your account has been enabled to use VASP (see the license conditions in the Access Prerequisite section) you need to decide on the version of the software you want to use. Use 

$ module avail vasp

to check what versions are available. We normally recommend using the latest version available. For example, to load the version 5.4.4 of vasp use 

$ module load vasp/5.4.4

For more details on using modules see our modules help guide.

VASP runs under the batch system by using a job script similar to the following example file jobscript:

#!/bin/bash

#PBS -l walltime=1:00:00
#PBS -l mem=10gb
#PBS -l ncpus=8
#PBS -l software=vasp
#PBS -l wd

# Load module, always specify version number.
module load vasp/5.4.4

# Must include `#PBS -l storage=scratch/ab12+gdata/yz98` if the job
# needs access to `/scratch/ab12/` and `/g/data/yz98/`. Details on:
# https://opus.nci.org.au/display/Help/PBS+Directives+Explained

mpirun vasp_std >vasp.log

To submit the job to PBS run the command

$ qsub jobscript

...
rcache.c:896 UCX ERROR failed to insert region 0x37bdc320 [0x1517ace43180..0x151769af8ac0]: Invalid parameter
...

...
mpirun -mca coll_hcoll_enable 0 vasp_std >vasp.log

Details of specific versions:

v5.4.4

The following extensions are included in each vasp executable:

• Vasp TST tools (http://theory.cm.utexas.edu/vtsttools/) v179
• Solvation: http://vaspsol.mse.ufl.edu/  v1.0
• Wannier90: http://www.wannier.org/ v3.0.0

Name of the vasp executables:

• vasp_std - this is normal version of vasp.
• vasp_ncl - is build without NGZhalf flag for spin-orbit coupling calculations.
• vasp_gam - gamma only version of vasp.
• vasp_std-x - vasp_std with cell optimization constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
• vasp_ncl-x - vasp_ncl with cell optimization constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
• vasp_gam-x - vasp_gam with cell optimization constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
• vasp_gpu and vasp_gpu_ncl are GPU versions of vasp_std and vasp_ncl.

An example of PBS script to run vasp 5.4.4 on GPUs. 

#!/bin/bash

#PBS -q gpuvolta
#PBS -l walltime=1:00:00
#PBS -l mem=10gb
#PBS -l ncpus=12
#PBS -l ngpus=1
#PBS -l software=vasp
#PBS -l wd

mkdir $PBS_JOBFS/nvidia-mps
export CUDA_MPS_PIPE_DIRECTORY=$PBS_JOBFS/nvidia-mps
mkdir $PBS_JOBFS/nvidia-log
export CUDA_MPS_LOG_DIRECTORY=$PBS_JOBFS/nvidia-log
nvidia-cuda-mps-control -d

# Load module, always specify version number.
module load vasp/5.4.4

# Must include `#PBS -l storage=scratch/ab12+gdata/yz98` if the job
# needs access to `/scratch/ab12/` and `/g/data/yz98/`. Details on:
# https://opus.nci.org.au/display/Help/PBS+Directives+Explained

mpirun vasp_gpu > vasp.out

v6.1.0

The following extensions are included in each vasp executable:

• At the time of the build there were no version of VTST tools compatible to vasp v.6.1.0
• Solvation: http://vaspsol.mse.ufl.edu/  v1.0
• Wannier90: http://www.wannier.org/ v3.1.0

Everything else is similar to vasp v5.4.4.

v6.2.1

The following extensions are now included in each vasp executable:

• Vasp TST tools (http://theory.cm.utexas.edu/vtsttools/) v182
• Solvation: http://vaspsol.mse.ufl.edu/  v1.0
• Wannier90: http://www.wannier.org/ v3.1.0

Name of the vasp executables:

• vasp_std - this is normal version of vasp.
• vasp_ncl - is build without NGZhalf flag for spin-orbit coupling calculations.
• vasp_gam - gamma only version of vasp.
• vasp_std-x - vasp_std with cell optimization constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
• vasp_ncl-x - vasp_ncl with cell optimization constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
• vasp_gam-x - vasp_gam with cell optimization constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
• vasp_std-gpu, vasp_ncl-gpu and vasp_gam-gpu are GPUs versions of vasp. Only vasp TST tools are included in the GPU version.

Starting from version 6.2.1 VASP introduced completely new GPU build. This runs much faster then the old GPU version, easily doing 8-10x times speedup in SCF in our limited testing.

There are now 3 GPU executables: vasp_sdt-gpu, vasp_ncl-gpu and vasp_gam-gpu. Then can only run in gpuvolta queue. Note that only Vasp TST is included into GPU version, vaspsol and wannier90 are not.

When you are using GPU versions of VASP, the limiting factor is GPU memory, the program fails if it needs more memory than GPU has. Note that this memory is fixed and is different from what you request in #PBS -l mem= line. You will need to request more GPUs so the system will fit into GPUs memory.

Below are two examples of PBS scripts for VASP GPU versions for 1 node (4 GPUs) and 2 nodes (8 GPUs):

1 node VASP GPU example

#!/bin/bash

#PBS -q gpuvolta
#PBS -l walltime=10:00:00
#PBS -l ncpus=48
#PBS -l ngpus=4
#PBS -l mem=160GB
#PBS -l jobfs=1GB
#PBS -l wd

# Load module, always specify version number.
module load vasp/6.2.1

# Must include `#PBS -l storage=scratch/ab12+gdata/yz98` if the job
# needs access to `/scratch/ab12/` and `/g/data/yz98/`. Details on:
# https://opus.nci.org.au/display/Help/PBS+Directives+Explained

mpirun -np $PBS_NGPUS --map-by ppr:1:numa vasp_std-gpu >vasp.log

2 nodes VASP GPU example

#!/bin/bash

#PBS -q gpuvolta
#PBS -l walltime=10:00:00
#PBS -l ncpus=96
#PBS -l ngpus=8
#PBS -l mem=320GB
#PBS -l jobfs=2GB
#PBS -l wd

# Load module, always specify version number.
module load vasp/6.2.1

# Must include `#PBS -l storage=scratch/ab12+gdata/yz98` if the job
# needs access to `/scratch/ab12/` and `/g/data/yz98/`. Details on:
# https://opus.nci.org.au/display/Help/PBS+Directives+Explained

mpirun -np $PBS_NGPUS --map-by ppr:1:numa vasp_std-gpu >vasp.log

VASP v6.2.1 CPU build can now also run with OMP, an example PBS script is below is using 2 nodes and 12 OMP threads:

#!/bin/bash

#PBS -q normal
#PBS -l walltime=10:00:00
#PBS -l ncpus=96
#PBS -l mem=320GB
#PBS -l jobfs=2GB
#PBS -l wd

# Load module, always specify version number.
module load vasp/6.2.1

# Must include `#PBS -l storage=scratch/ab12+gdata/yz98` if the job
# needs access to `/scratch/ab12/` and `/g/data/yz98/`. Details on:
# https://opus.nci.org.au/display/Help/PBS+Directives+Explained

export OMP_NUM_THREADS=12
export OMP_PLACES=cores
export OMP_PROC_BIND=TRUE

mpirun -np $(($PBS_NCPUS/$OMP_NUM_THREADS)) --map-by ppr:1:numa:PE=12 vasp_std >vasp.log

In our limited testing we did not see much difference between MPI and MPI/OMP versions, but this is likely to be system dependent.

Please, check VASP OMP notes in the VASP user manual on the vasp web site for more details.

v6.3.1

The following extensions are now included in each vasp executable:

• Vasp TST tools (http://theory.cm.utexas.edu/vtsttools/) v184
• Solvation: http://vaspsol.mse.ufl.edu/  v1.0
• Wannier90: http://www.wannier.org/ v3.1.0
• hdf5 v1.12.1

Name of the vasp executables:

• vasp_std - this is normal version of vasp.
• vasp_ncl - is build without NGZhalf flag for spin-orbit coupling calculations.
• vasp_gam - gamma only version of vasp.
• vasp_std-x - vasp_std with cell optimization constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
• vasp_ncl-x - vasp_ncl with cell optimization constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
• vasp_gam-x - vasp_gam with cell optimization constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
• vasp_std-gpu, vasp_ncl-gpu and vasp_gam-gpu are GPUs versions of vasp. Only vasp TST tools are included in the GPU version.

This is the first vasp build that includes hdf5 library. As the result, quite a large file vaspout.h5 containing various summaries of the vasp calculations is created during the run.

Running vasp v6.3.1 is similar to vasp v6.2.1 with one very important exception when running on GPU nodes. An extra -mca coll ^hcoll parameter should be added to the mpirun command. For example:

2 nodes VASP GPU example

#!/bin/bash

#PBS -q gpuvolta
#PBS -l walltime=10:00:00
#PBS -l ncpus=96
#PBS -l ngpus=8
#PBS -l mem=320GB
#PBS -l jobfs=2GB
#PBS -l wd

# Load module, always specify version number.
module load vasp/6.3.1

# Must include `#PBS -l storage=scratch/ab12+gdata/yz98` if the job
# needs access to `/scratch/ab12/` and `/g/data/yz98/`. Details on:
# https://opus.nci.org.au/display/Help/PBS+Directives+Explained

mpirun -np $PBS_NGPUS --map-by ppr:1:numa -mca coll ^hcoll vasp_std-gpu >vasp.log

License requirements

Users who wish to use VASP5 must have an existing VASP5 license with Prof Juergen Hafner (https://www.vasp.at/). The holder of the license needs to add users from the group to the list of users at https://www.vasp.at. Once this is done a user from this list can request to be added to vasp5 and/or vasp6 group. We will check the users email in the list and allow access if the user is registered at https://www.vasp.at. Note that email addresses in the NCI and https://www.vasp.at/ should be the same.