On NCI compute systems, the following debug tool can be used:

GDB, the GNU Project debugger, allows you to see what is going on `inside' another program while it executes -- or what another program was doing at the moment it crashed.

More information: https://www.gnu.org/software/gdb/

Debugging without coredump files

$ cc -g prog.c
$ gdb ./a.out
...
(gdb) list
(gdb) stop at 10
(gdb) run
(gdb) print variable
...
(gdb) quit

Debugging with coredump files

By default your jobs will not produce coredump files when they crash. To generate corefiles you need:

$ ulimit -c unlimited            # For bash
$ limit coredumpsize unlimited   # For tcsh

Also if you are using the Intel Fortran compiler, you will will need:

$ export decfort_dump_flag=y    # For bash
$ setenv decfort_dump_flag y    # For tcsh

To use the coredump file, enter:

$ cc -g prog.c
$ gdb
$ gdb ./a.out /path/to/the/coredumpfile
...
(gdb) where
(gdb) bt
(gdb) frame number
(gdb) list
(gdb) info locals
(gdb) print variable
...
(gdb) quit

Coredump files can take up a lot of disk space – be careful not to generate them if you are not going to use them.