Page tree
Skip to end of metadata
Go to start of metadata

The Vampir suite is a set of tools for profiling MPI, multi-threaded and serial applications. VampirTrace itself is a tool that can be used to produce execution traces in OTF (Open Trace Format).

This page describes the use of VampirTrace to instrument code and generate OTF execution traces. To analyze execution traces, use the Vampir tool; for details, see the software map entry for that tool.

Note that although VampirTrace and Intel Trace Collector are essentially different forks of the same software, they store traces in incompatible trace formats. Specifically, ITC stores traces in STF (Structured Trace Format), a proprietary format owned by Intel that is not compatible with OTF.


There are two phases to producing an OTF trace using VampirTrace. The first phase is to instrument your code and produce a VampirTrace-enabled application binary; the second phase is to perform an execution run and actually produce the trace. These two phases will be covered separately below.

As a preliminary, note that VampirTrace is currently only available as a bundled component within OpenMPI modules with version number 1.3.0 or later.

Instrumenting code for VampirTrace

As mentioned above, to gain access to VampirTrace, you will need to load an OpenMPI module appropriate to your choice of compiler. For example, 

module load openmpi/1.4.3

(Of course, from a profiling perspective, it makes the most sense to use the same version of OpenMPI that you have been using already.)

VampirTrace can automatically instrument your code. (It is also possible to manually instrument code; for further details, refer to the VampirTrace user manual.) To do so, the build process for your code must be repeated, passing all compilation and linking commands through VampirTrace compiler wrappers. As build systems and scripts differ across applications, it is impossible to give any more than broad instructions here; if you experience problems, consult Section 2.1 of the VampirTrace manual, and if pain persists, please email NCI help.

VampirTrace provides several basic compiler wrappers: vtcc, vtcxx, vtf77, and vtf90. These wrap the standard compilers for C, C++, Fortran77 and Fortran90 respectively. Additionally, VampirTrace provides MPI-enabled wrappers: mpicc-vt, mpicxx-vt, mpif77-vt, and mpif90-vt. These are equivalent to the standard MPI wrappers that come with OpenMPI (mpicc, mpicxx,mpif77, and mpif90), except that they also enable automatic instrumentation. In many/most cases, using these wrappers should be as simple as replacing a makefile line like

CC = mpicc

with a line like

CC = mpicc-vt

When you have modified your build script, re-run it. The resultant application binary/binaries should be VampirTrace-enabled, and ready to produce an execution trace.

 Producing an OTF trace from instrumented code

Once you have successfully instrumented your code with VampirTrace, all that remains to be done is to execute the code and produce a trace. When an instrumented binary is run, VampirTrace reads several parameters at runtime, which are set by exporting particular environment variables with the prefix VT_ (e.g. VT_FILE_PREFIX). For a full description of parameters accepted, refer to the VampirTrace user manual.

A sample PBS script follows for collecting an execution trace from a VampirTrace-instrumented binary named fft.

#! /bin/bash

# Sample PBS script for producing an OTF trace from a VampirTrace-instrumented
# binary. 

#PBS -q normal
#PBS -l walltime=00:03:00
#PBS -l mem=8GB
#PBS -l ncpus=32

# Load OpenMPI v1.3.0 or greater to get access to VampirTrace.
module load intel-cc
module load openmpi/1.4.3

# Move into the working directory, i.e. where the script
# was submitted from. We assume from hereon in that this
# is where the target application resides.

# Define a name for the trace.
export TRACE_NAME=fft

# Create a directory to store the trace. Here, we will create "fft_trace".
export TRACE_DIR_SUFFIX=_trace
rm -rf $TRACE_DIR
mkdir $TRACE_DIR

# Tell VampirTrace to use the created directory, and specified trace name.

# Use nm to create a BSD-format symbol list for the application executable.
# We tell nm to try and find the line numbers of symbol definitions from
# debug info; VampirTrace can integrate this information into the trace.
nm --format=bsd --line-numbers fft > fft.nm

# Tell the system where to find the symbol list file.
export VT_NMFILE=$PBS_O_WORKDIR/fft.nm

# Run the executable, and (hopefully!) extract a valid trace.
mpirun ./fft 

Some points raised by the above script:

  • Use of nm: Ordinarily, VampirTrace would use the nm utility to extract symbol information behind the scenes. Check that this works. Failure to provide an appropriate nm symbol-file output will result in your binary failing to execute. (If you wish, you could integrate the creation of this file within your build script; however, care must be exercised to ensure that this symbol list remains consistent with the application binary.)
  • Checking for unification: VampirTrace will normally "unify" trace information after execution. In some cases, this will fail, and must be done manually; this script uses the absence of a master trace file as evidence of failed automatic unification. For more information about manual unification, see "Additional Notes" below.
  • Creation of a tarball: As described in "Additional Notes" below, an OTF trace consists of multiple files. For the sake of safety, it is recommended to store your trace in a way that minimises the possibility of losing a file. Using a tarball helps in this, but is not necessary.

Once you have extracted a trace, you can view and analyze it using Vampir -- refer to the software map entry for that tool.

Additional Notes

OTF Trace Structure

An OTF trace is composed of at least three and potentially many files. It is important to keep these files together -- one missing file will invalidate a trace. (For this reason, it is a good idea to segregate files for a single trace in a separate directory.) For a given trace name, corresponding to an execution run that utilised n_procs MPI processes, the trace will consist of the files:

  • One file with name <tracename>.otf. This is the master trace file.
  • One file with name <tracename>.0.def.z. This is the trace definition file.
  • n_procs files with names <tracename>.<i_hex>.events.z. Each of these files corresponds to the trace information from a single MPI process. i_hex is the ith process number, expressed in hexadecimal.

The presence of files with the the extension .uctl, along with the lack of a master trace file, indicates that the trace has not been properly unified. See below for instructions on how to fix this. 

Manual Trace Unification

To manually unify a trace, load the appropriate openmpi module, navigate into the directory containing the various trace files, and execute

> vtunify <n_procs> <tracename>