Finding memory problems in code can be a difficult task but there are tools available on the NCI machines to make it possible. Memory errors can arise in many ways such as referencing arrays out of their declared bounds, failing to allocate dynamic arrays, attempting to free memory that cannot be freed or memory leaks leading to an increase in the memory requirements of the running program.

• Debuggers such as gdb can be used to pinpoint lines where segmentation violations occur. However this may not be the actual point at which the error occured. The difficulty with memory problems is that the error may be reported some time after the incorrect line of code. This requires that the code is compiled with the -g option.
• For Fortran code it is always worth compiling with the option -check bounds to pinpoint any array subscript or substring references that are out of declared bounds at runtime.
• Electric Fence can be used to detect errors in C code where the boundaries of a malloc() are overrun or where there is an attempt to touch a memory allocation that has already been freed. Although the Electric Fence documentation will say that it can be used to debug MPI codes this does not work on the AC. It appears that the start-up procedure for the executable linked with -lefence invokes some memory allocations that interfere with the SGI mpirun start-up. Code must be linked with the libefence.a library before being executed.

• Mudflap is part of the more recent versions of gcc/g++. To use mudflap you need to do at least

  $ gcc prog.c -fmudflap -lmudflap $ a.out

  The environment variable MUDFLAP_OPTIONS can be used to control the output from mudflap. See http://gcc.gnu.org/wiki/Mudflap_Pointer_Debugging for more details.

• TotalView: https://opus.nci.org.au/display/Help/TotalView

• Intel Inspector: https://software.intel.com/en-us/intel-inspector-xe
• DrMemory: https://github.com/dynamorio/drmemory
• GNU mtrace: http://www.gnu.org/software/libc/manual/html_node/Tracing-malloc.html