Page tree

Overview

VASP stands for Vienna Ab-initio Simulation Package, and is a package for performing ab-initio quantum-mechanical molecular dynamics (MD) using pseudopotentials and a plane wave basis set.

VASP is distributed though Prof. Juergen Hafner. See the VASP web site for more details.

How to use 


Once your account has been enabled to use VASP (see the licence conditions in the Access Prerequisite section) you need to decide on the version of the software you want to use. Use 

 $ module avail vasp

to check what versions are available. We normally recommend using the latest version available. For example, to load the version 5.4.4 of vasp use 

 $ module load vasp/5.4.4

For more details on using modules see our software applications guide.

VASP runs under the batch system by using a job script similar to the following example file jobscript:

#!/bin/bash
 
#PBS -l walltime=1:00:00
#PBS -l mem=10gb
#PBS -l ncpus=8
#PBS -l software=vasp
#PBS -l wd
 
# Load module, always specify version number.
module load vasp/5.4.4
 
# Must include `#PBS -l storage=scratch/ab12+gdata/yz98` if the job
# needs access to `/scratch/ab12/` and `/g/data/yz98/`
 
mpirun vasp_std >vasp.log

To submit the job to PBS run the command

$ qsub jobscript
POTCAR files for vasp in /apps/vasp/potpaw_LDA.54 and /apps/vasp/potpaw_PBE.54 directories.
Vasp is built with MKL scalapack. 
Please, pay particular attention to NPAR (or NCORE) parameter. This parameter determines how well vasp will run in parallel.  The recommended value should be somewhere between SQRT(ncpus) ... ncpus/2, closer to SQRT( ncpus ).  For example, for ncpus=64, the npar=8 or 16 is good.

If your job is failed with an error like

...
rcache.c:896 UCX ERROR failed to insert region 0x37bdc320 [0x1517ace43180..0x151769af8ac0]: Invalid parameter
...

please modify your job script like the following way to fix the issue:

...
mpirun -mca coll_hcoll_enable 0 vasp_std >vasp.log

Details of specific versions


v5.4.4 

The following extensions are included in each vasp executable:

Name of the vasp executables:

  • vasp_std - this is normal version of vasp.
  • vasp_ncl - is build without NGZhalf flag for spin-orbit coupling calculations.
  • vasp_gam - gamma only version of vasp.
  • vasp_std-x - vasp_std with cell optimization constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
  • vasp_ncl-x - vasp_ncl with cell optimization constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
  • vasp_gam-x - vasp_gam with cell optimization constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
  • vasp_gpu and vasp_gpu_ncl are GPU versions of vasp_std and vasp_ncl.

An example of PBS script to run vasp 5.4.4 on GPUs. 

#!/bin/bash
 
#PBS -q gpuvolta
#PBS -l walltime=1:00:00
#PBS -l mem=10gb
#PBS -l ncpus=12
#PBS -l ngpus=1
#PBS -l software=vasp
#PBS -l wd
 
mkdir $PBS_JOBFS/nvidia-mps
export CUDA_MPS_PIPE_DIRECTORY=$PBS_JOBFS/nvidia-mps
mkdir $PBS_JOBFS/nvidia-log
export CUDA_MPS_LOG_DIRECTORY=$PBS_JOBFS/nvidia-log
nvidia-cuda-mps-control -d
 
# Load module, always specify version number.
module load vasp/5.4.4
 
# Must include `#PBS -l storage=scratch/ab12+gdata/yz98` if the job
# needs access to `/scratch/ab12/` and `/g/data/yz98/`
 
mpirun vasp_gpu > vasp.out


There are several limitations in vasp v5.4.4 gpu build. Check vasp web site for details.

v6.1.0 

The following extensions are included in each vasp executable:

Everything else is similar to vasp v5.4.4.

v6.2.1 

The following extensions are now included in each vasp executable:

Name of the vasp executables:

  • vasp_std - this is normal version of vasp.
  • vasp_ncl - is build without NGZhalf flag for spin-orbit coupling calculations.
  • vasp_gam - gamma only version of vasp.
  • vasp_std-x - vasp_std with cell optimisation constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
  • vasp_ncl-x - vasp_ncl with cell optimisation constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
  • vasp_gam-x - vasp_gam with cell optimisation constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
  • vasp_std-gpu, vasp_ncl-gpu and vasp_gam-gpu are GPUs versions of vasp. Only vasp TST tools are included in the GPU version.

Starting from version 6.2.1 VASP introduced completely new GPU build. This runs much faster then the old GPU version, easily doing 8-10x times speedup in SCF in our limited testing.

There are now 3 GPU executables: vasp_sdt-gpu, vasp_ncl-gpu and vasp_gam-gpu. Then can only run in gpuvolta queue. Note that only Vasp TST is included into GPU version, vaspsol and wannier90 are not.

When you are using GPU versions of VASP, the limiting factor is GPU memory, the program fails if it needs more memory than GPU has. Note that this memory is fixed and is different from what you request in #PBS -l mem= line. You will need to request more GPUs so the system will fit into GPUs memory.

Below are two examples of PBS scripts for VASP GPU versions for 1 node (4 GPUs) and 2 nodes (8 GPUs):

1 node VASP GPU example  

#!/bin/bash
 
#PBS -q gpuvolta
#PBS -l walltime=10:00:00
#PBS -l ncpus=48
#PBS -l ngpus=4
#PBS -l mem=160GB
#PBS -l jobfs=1GB
#PBS -l wd
 
# Load module, always specify version number.
module load vasp/6.2.1
 
# Must include `#PBS -l storage=scratch/ab12+gdata/yz98` if the job
# needs access to `/scratch/ab12/` and `/g/data/yz98/`
 
mpirun -np $PBS_NGPUS --map-by ppr:1:numa vasp_std-gpu >vasp.log

2 nodes VASP GPU example 

#!/bin/bash
 
#PBS -q gpuvolta
#PBS -l walltime=10:00:00
#PBS -l ncpus=96
#PBS -l ngpus=8
#PBS -l mem=320GB
#PBS -l jobfs=2GB
#PBS -l wd
 
# Load module, always specify version number.
module load vasp/6.2.1
 
# Must include `#PBS -l storage=scratch/ab12+gdata/yz98` if the job
# needs access to `/scratch/ab12/` and `/g/data/yz98/`

mpirun -np $PBS_NGPUS --map-by ppr:1:numa vasp_std-gpu >vasp.log

Note that scripts are using PBS_NGPUS for the number of MPI processes, not PBS_NCPUS you would normally use in CPU version of vasp.


VASP v6.2.1 CPU build can now also run with OMP, an example PBS script is below is using 2 nodes and 12 OMP threads:

#!/bin/bash
 
#PBS -q normal
#PBS -l walltime=10:00:00
#PBS -l ncpus=96
#PBS -l mem=320GB
#PBS -l jobfs=2GB
#PBS -l wd
 
# Load module, always specify version number.
module load vasp/6.2.1
 
# Must include `#PBS -l storage=scratch/ab12+gdata/yz98` if the job
# needs access to `/scratch/ab12/` and `/g/data/yz98/`
 
export OMP_NUM_THREADS=12
export OMP_PLACES=cores
export OMP_PROC_BIND=TRUE
 
mpirun -np $(($PBS_NCPUS/$OMP_NUM_THREADS)) --map-by ppr:1:numa:PE=12 vasp_std >vasp.log

In our limited testing we did not see much difference between MPI and MPI/OMP versions, but this is likely to be system dependent.

Please, check VASP OMP notes in the VASP user manual on the vasp web site for more details.

v6.3.1 

The following extensions are now included in each vasp executable:

Name of the vasp executables:

  • vasp_std - this is normal version of vasp.
  • vasp_ncl - is build without NGZhalf flag for spin-orbit coupling calculations.
  • vasp_gam - gamma only version of vasp.
  • vasp_std-x - vasp_std with cell optimisation constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
  • vasp_ncl-x - vasp_ncl with cell optimisation constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
  • vasp_gam-x - vasp_gam with cell optimisation constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
  • vasp_std-gpu, vasp_ncl-gpu and vasp_gam-gpu are GPUs versions of vasp. Only vasp TST tools are included in the GPU version.

This is the first vasp build that includes hdf5 library. As the result, quite a large file vaspout.h5 containing various summaries of the vasp calculations is created during the run.

Running vasp v6.3.1 is similar to vasp v6.2.1 with one very important exception when running on GPU nodes. An extra -mca coll ^hcoll parameter should be added to the mpirun command. For example:

2 nodes VASP GPU example

#!/bin/bash
 
#PBS -q gpuvolta
#PBS -l walltime=10:00:00
#PBS -l ncpus=96
#PBS -l ngpus=8
#PBS -l mem=320GB
#PBS -l jobfs=2GB
#PBS -l wd
 
# Load module, always specify version number.
module load vasp/6.3.1
 
# Must include `#PBS -l storage=scratch/ab12+gdata/yz98` if the job
# needs access to `/scratch/ab12/` and `/g/data/yz98/`
 
mpirun -np $PBS_NGPUS --map-by ppr:1:numa -mca coll ^hcoll vasp_std-gpu >vasp.log

v6.3.2 

The following extensions are now included in each vasp executable:

Name of the vasp executables:

  • vasp_std - this is normal version of vasp.
  • vasp_ncl - is build without NGZhalf flag for spin-orbit coupling calculations.
  • vasp_gam - gamma only version of vasp.
  • vasp_std-x - vasp_std with cell optimisation constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
  • vasp_ncl-x - vasp_ncl with cell optimisation constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
  • vasp_gam-x - vasp_gam with cell optimisation constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
  • vasp_std-gpu, vasp_ncl-gpu and vasp_gam-gpu are GPUs versions of vasp. Only vasp TST tools are included in the GPU version.

Running vasp v6.3.2 is similar to vasp v6.3.1.

v6.4.2 

The following extensions are now included in each vasp executable:

Name of the vasp executables:

  • vasp_std - this is normal version of vasp.
  • vasp_ncl - is build without NGZhalf flag for spin-orbit coupling calculations.
  • vasp_gam - gamma only version of vasp.
  • vasp_std-x - vasp_std with cell optimisation constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
  • vasp_ncl-x - vasp_ncl with cell optimisation constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
  • vasp_gam-x - vasp_gam with cell optimisation constrained along X axis only. Also available are -z and -xy versions, constraining Z axis and XY plane.
  • vasp_std-gpu, vasp_ncl-gpu and vasp_gam-gpu are GPUs versions of vasp. Only vasp TST tools are included in the GPU version.

Running vasp v6.4.2 is similar to vasp v6.3.2.

Licence requirements 


Users who wish to use VASP5 must have an existing VASP5 licence with Prof Juergen Hafner https://www.vasp.at/. The holder of the licence needs to add users from the group to the list of users at https://www.vasp.at. Once this is done a user from this list can request to be added to vasp5 and/or vasp6 group. We will check the users email in the list and allow access if the user is registered at https://www.vasp.at. Note that email addresses in the NCI and https://www.vasp.at/ should be the same.

There are two different groups in MyNCI: https://my.nci.org.au/mancini/project/vasp5/join for vasp5 and https://my.nci.org.au/mancini/project/vasp6/join for vasp6. The vasp5 group grants access to vasp v5.4.4 on Gadi. The vasp6 groups allows to use vasp v6. Users are entitled for vasp6 access only if they are in the licence  with a “Valid to” date later than 30 June 2019. If you are not sure what is the "Valid to" date of your licence, we can check it. Just send request to our helpdesk.

Authors: Yue Sun, Andrey Bliznyuk, Mohsin Ali
  • No labels