Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

This document describes how to use MPI performance analysis tools including profilers and tracers which are available on NCI NF compute NCI compute systems. For further help with using MPI profilers and tracers, please send email to help@nci.org.au .

 

...

Table of Contents

MPI Profiling

...

  1. Usage
  2. Profile Data
  3. Graphical Parser and Viewer
  4. Integration with Hardware Performance Counters (HPM)
  5. NCI Customized IPM Settings

...

  1. Usage
  2. Profile Data
  3. Graphical Viewer

...

  1. Usage

...

 

MPI PROFILING

A MPI profiler aggregates “whole run” statistics at run time , e.g. , total amount of time spent in MPI, total number of messages or bytes sent, etc. As this information is available on a per-rank basis, issues such as load imbalance are exposed.   

...

Note that (currently) no profiling information will be produced if the execution does not complete normally (i.e. does not call MPI_Finalize()).   

On NCI NF compute systems, two different lightweight MPI profilers are installed. They are IPM and mpiP. Both of these tools require minimal actions to invoke – we recommend that you use them regularly. Note that their use is only applicable to Open MPI applications.   

IPM

IPM is a widely-used MPI profiler with many sites using it by default for every MPI job. The level of detail is selectable at runtime and presented through a variety of text and web reports. IPM can also provide information from the CPU HW counters giving basic CPU as well as MPI statistics.   

For version The following versions of IPM are available:

  1. ipm/0.983-nci
    1. works with openmpi/1.6.5

...

    1.  or less
    2. gives nice graph with communication pattern
  1. ipm/0.983-

...

  1. cache
    1. works with openmpi/1.6.5 or less
    2. gives L1 L2 L3 cache misses
  2. ipm/2.0.2

...

    1. only works with openmpi/1.7.* and 1.8.*
    2. the only version gives flops
  1. ipm/2.0.5
    1. works with openmpi/1.10.2

Usage

Using IPM does not require code recompilation. Instead,

...

 LD_

...

PRELOAD is used to dynamically load the IPM library (libipm.so) as a wrapper to the MPI runtime.   

A simple PBS job script using the IPM profiler with an MPI executable (prog.exe) is as shown below:   

...

Code Block
languagebash
themeRDark
#!/bin/

...

bash  
#PBS -l ncpus=

...

2
module load

...

 openmpi
module

...

 load ipm
mpirun prog.exe >

...

 output

Currently, IPM is available for openmpi version 1.4.1 and above.   

User can also define the IPM log directory and log file name via setting following environment variables in the PBS job script before mpirun, an good example of IPM log directory and file name are:   

Code Block
languagebash
themeRDark
export IPM_LOGDIR=/short/$PROJECT/ipm_logs
export IPM_LOGFILE=$PBS_JOBID.$USER.$PROJECT.`date +%s`

NOTE: For some applications, defining IPM_LOGDIR in .bashrc or .cshrc is compulsory to successfully generate IPM profile data file. These applications usually have following features:   

  • different MPI ranks working in different directories,
  •   
  • PBS job scripts are automatically generated by some sense,
  •   
  • more to be added

    …     

...

Profile Data

By default IPM produces a summary of the performance information for the application on stdout. IPM also generates an XML data file which will be named something like (if user hasn’t define IPM_LOGFILE

...

environment variable):   

your_username.1231369287.321103.0 eg. jxc900.1231369287.321103.0  

...

Graphical Parser and Viewer

The XML data file can be used to generate a graphical webpage in one of two ways.   

Use lightweight browser on NCI

...

machine

To visualize the IPM XML data on Raijin, you need to login to Raijin with an X display, eg. using ssh -X or ssh -Y, or with VNC. The detailed sample instruction on Raijin is listed below.   

...

Code Block
languagebash
themeRDark
ssh -X raijin

...

module load openmpi

...

module load ipm

...

ipm_view IPM_XML_file

 

Use your favourite browser on your laptop/desktop

Alternatively, the IPM XML data file can be parsed to HTML format. The detailed sample instruction on raijin are:   

...

Code Block
languagebash
themeRDark
ssh raijin

...

module load openmpi

...

module load ipm

...

module load ploticus

...

ipm_parse -html <IPM_XML_file>

The ipm_parse command will generate a directory containing parsed IPM profile data with graphs. The directory will be named something like:   

a.out_1_your_username.1231369287.321103.0_ipm_${jobid}  

You can secure copy the directory to your local disk. A sample instruction is as follows.   

...

Code Block
languagebash
themeRDark
scp -r a.out_1_your_username.1231369287.321103.0_ipm_${jobid} user@your_local_machine:path_to_store/

Then you can view it with your favourite web browser, e.g. firefox, on your desktop:   

...

Code Block
languagebash
themeRDark
firefox path_to_store/a.out_1_your_username.1231369287.321103.0_ipm_${jobid}/index.html

 

...

Integration with Hardware Performance Counters (HPM)

IPM can be integrated with hardware performance counters and profile useful information such as GFlops, cache misses, etc. PAPI is used for this purpose. Currently, IPM-HPM are only avaialble for OpenMPI Open MPI version 1.4.3.   

To usage IPM with HPM, please do following:   

Code Block
languagebash
themeRDark
module load ipm/0.983-

...

cache

A sample PBS job script will be similar to following:   

...

Code Block
languagebash
themeRDark
#!/bin/bash

...

#PBS -l ncpus=2

...

module load openmpi

...

module load ipm/0.983-cache

...

export IPM_HPM=PAPI_FP_OPS,PAPI_TOT_INS,PAPI_TOT_CYC,PAPI_L2_TCA

...

mpirun prog.exe > output

 

Pre-defined PAPI Eventsets

The environment variable IPM_HPM can be selected from following predefined PAPI event sets for Raijin.   

For Raijin:  

...

  1. PAPI_FP_OPS,PAPI_TOT_INS,PAPI_TOT_CYC,PAPI_L2_TCA

...

  1. PAPI_FP_OPS,PAPI_TOT_INS,PAPI_TOT_CYC,PAPI_L2_TCA,PAPI_L2_TCM

...

  1. PAPI_TOT_INS,PAPI_TOT_CYC,PAPI_L2_STM,PAPI_L2_LDM,PAPI_L2_TCM,PAPI_L2_DCM

...

  1. PAPI_TOT_CYC,PAPI_TOT_INS,PAPI_L3_TCM,PAPI_L3_LDM,PAPI_SR_INS,PAPI_LD_INS

...

  1. MEM_LOAD_RETIRED:OTHER_CORE_L2_HIT_HITM,MEM_UNCORE_RETIRED:OTHER_CORE_L2_HITM

...

  1. PAPI_FP_OPS,PAPI_FP_INS,PAPI_DP_OPS,PAPI_VEC_DP

...

  1. PAPI_FP_OPS,PAPI_FP_INS,PAPI_SP_OPS,PAPI_VEC_SP

...

  1. PAPI_FP_OPS,PAPI_RES_STL,PAPI_TOT_CYC,PAPI_TOT_INS

...

  1. PAPI_FP_OPS,PAPI_TLB_DM,PAPI_TLB_IM

...

  1. PAPI_L1_DCA,PAPI_L2_DCM

 

Customize Your Own PAPI Eventsets

You can also customize your own PAPI eventsets for IPM_HPM environment variable via using papiusing papi_event_chooser commandchooser command  

...

Code Block
languagebash
themeRDark
module load papi

...

papi_event_chooser -help

Both PAPI pre-define events, and native hardware events can be used.   

NCI

...

Customised IPM

...

Settings

 

Message sizes rounding to power of 2

...

Code Block
languagebash
themeRDark
export IPM_ROUNDED

...

=yes

This feature significantly reduces the memory usage and postprocessing time during both runtime and parsing time.   

IPM Check-pointing

...

Code Block
languagebash
themeRDark
export IPM_CHECKPOINT

...

=yes

...

export IPM_CHKPT_INTERVAL

...

=3600

This feature allow IPM to checkpoint profile status in $IPM_CHKPT_INTERVAL seconds intervals. The checkpointed profile data will be stored in a.out_1_your_username.1231369287.321103.0_ipm_${jobid}.${rank} files.   

These files can be merged (using cat commandusing cat command) into a single xml XML file, and parsed with ipm_parse  

...

mpiP

mpiP is a lightweight profiling library for MPI applications. In addition to the MPI summary profiling provided by IPM, mpiP can provide “call site” statistics showing which calls in the code are dominating MPI execution time.   

...

Usage

Using mpiP does not require code recompilation. Instead,

...

 LD_

...

PRELOAD is used to dynamically load the mpiP library (libmpiP.so) as a wrapper to the MPI runtime.   

A sample PBS job script using mpiP profiler with MPI executable (prog.exe) is as shown below:   

...

Code Block
languagebash
themeRDark
#!/bin/bash

...

#PBS -l ncpus=2

...

module load openmpi

...


...

module load mpiP

...

mpirun prog.exe > output

Currently, mpiP is available for openmpi version 1.3.3 and above.   

...

Profile Data

The mpiP profiler generates a text based output file named something like:   

prog.exe.${np}.25972.1.mpiP  

...

Graphical Viewer

To

...

visualise the mpiP profile data on Raijin, you need to login to Raijin with a X display, eg. using ssh -X or ssh -Y, or with VNC. The detailed sample instruction on raijin is listed below.

...

Code Block
languagebash
themeRDark
ssh -X raijin

...

module load openmpi

...

module load mpiP

...

mpirun prog.exe
mpipview prog.exe.${np}.XXXXX.1.mpiP

mpipview is able to map MPI callsites in the profile data to source code. This requires the MPI program to be compiled with -g option and linked with libunwindwith libunwind, as follows.   

...

Code Block
languagebash
themeRDark
module load openmpi

...

module load mpiP

...

mpicc -g -o prog.exe prog.c -lmpiP -lm -lbfd -liberty -lunwind

...

 

COORPERATION WITH GENERAL PROFILERS


mpirun prog.exe

Cooperation with General Profilers

Due to MPI profilers only profile for MPI function calls, it is not sufficient to reveal other details of the application. To get a better knowledge of users program, for example:   

  • which portion of the user program spent the most time,  
  • what is the memory behaviour of this program, including number of load/store instructions, cache misses, etc. 
  • how many bus transactions has been made in this program,   

...

Please refer  General profiling for the details of the general profilers available at NF NCI compute systems.   

MPI

...

Tracing

An MPI tracer collects an event history. It is common to display such event history on a timeline display. Tracing data can provide much interesting detail, but data volumes are large and the overhead of collection may be non-trivial. Often the collection of traces has to be limited in both duration and number of cpus to be feasible. The use of MPI tracing is strongly encouraged during the development or tuning of parallel applications but should not be used in production runs.   

MPI I/O

...

Profiling

Darshan  

DARSHAN  

USAGE

In raijin:   

In script of PBS,   

Usage

Use the following PBS script on Raijin

Code Block
languagebash
themeRDark
module load openmpi/1.6.5-mlx
module load darshan

...

export DARSHAN_LOG=./logdir # Log will be created under "./logdir" directory (make sure the directory is available)
/apps/openmpi/1.6.5-mlx/bin/mpicc ...
/apps/openmpi/1.6.5-mlx/bin/mpirun ...

After running, generate pdf PDF summary report from logs.   

darshan-job-summary.pl

...

./logdir/<JobID>***.darshan.gz

The NCI NF is investigating install the following MPI tracing tools:   

...

More information will be provided shortly.     

ALLINEA (TRIAL LICENSE)

The Allinea DDT&MAP and its performance report products are installed on Raijin for evaluation with a trial license.   

A brief homemade Usage Guide for Raijin (requested by Muhammad):   

  • login raijin with -X or -Y,  
  • module load allinea  
  • For DDT or MAP,  
cp /apps/allinea/4.2/tools/templates/pbs.qtf ~
edit pbs.qtf to be suitable for raijin
like adding #PBS -P z00 , #PBS -l mem 30GB
  • Compile your MPI program with “-g -fno-inline” for GNU, “-g -fno-inline-functions” for Intel compilers  
  • DDT or MAP  
    • Type “ddt” or “map” and click “Run” button  
    • Complete the Run window, something to remind,  
      • Implementation: choose openmpi with ticking “override default mpirun path” and using full path like “/apps/openmpi/1.6.5/bin/mpirun”  Make sure you tick “Memory Debugging” for DDT 

    • Tick “submit to queue”, click “configure” to select your template file, and Regexp for job id changed to “(\d+).r-man2”,  
    • Edit Queue parameters, edit the walltime and queue (to express or normal)  
    • Click “Submit” and you can debug/profile your program.  
  • perf-report  
    • Edit your PBS script with “module load allinea”  
    • Replace your mpirun command line as  perf-report /apps/openmpi/1.*/bin/mpirun .. (Note: full path of mpirun must be used)  

    • Then a HTML report will be generated (Note: it needs javascript so dillo doesn’t work. the best way may be copying it to your computer)   

More on usage can be found in raijin:/apps/allinea/4.2/tools/doc/userguide.pdf