NetCDF

On This Page

Overview

NetCDF (Network Common Data Form) is a set of software libraries and machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. It is also a community standard for sharing scientific data. For more information on using the NetCDF library, or any of the provided utilities, see the NetCDF documentation pages provided by Unidata

Usage

To use NetCDF on Gadi, first load one of the NetCDF modules.

$ module load netcdf/4.7.4

We always recommend specifying a version number when loading modules. In order to build an application on Gadi against NetCDF, only the NetCDF library explicitly needs to be linked.

$ gcc -o read_nc_file read_nc_file.c -lnetcdf

The libraries that NetCDF depends on are linked automatically by default using data stored in the shared NetCDF library, you do not need to load any underlying HDF5, SZip etc modules in order to compile your application.

To use the Fortran interface, the libnetcdff library also needs to be linked.

$ gfortran -o read_nf_file read_nf_file.f90 -lnetcdf -lnetcdff

Every NetCDF installation on Gadi has been built to be compatible with all GNU and Intel compilers available on Gadi. In the above commands, icc can be substituted for gcc and ifort can be substituted for gfortran without any other changes. The NetCDF libraries corresponding to your choice of compiler are selected at compile time by NCI's compiler wrappers. Every parallel NetCDF installation on Gadi is compatible with every MPI distribution installed on Gadi by a similar mechanism, with one minor exception:

When building your application with OpenMPI version 4.x, and a parallel NetCDF library, you must set the environment variable OMPI_MCA_io=ompio or pass --mca io ompio to mpirun when you run your application. This is due to a break in compatibility between the default MPI-IO implementation between OpenMPI 3 and 4, which are otherwise binary compatible.

Static linking

We do not usually recommend the use of static linking on Gadi, as it prevents applications from receiving important security or bug fixes that would otherwise be applied automatically when shared libraries are updated. If your application uses NetCDF and is compiled statically (e.g. by passing the --static flag to the compiler) then you will need to include the libraries that NetCDF depends on manually. The easiest way to do this is to load the appropriate modules and add the following nc-config command to your compiler flags:

$ gcc -static -o read_nc_file read_nc_file.c $( nc-config --libs --static )

Note that MPI applications cannot be built statically on Gadi.

Module Versions

There are two NetCDF modules available for each version (with the exception of NetCDF 4.6.3 at time of writing, which has three)

$ module avail netcdf
-------------------------- /apps/Modules/modulefiles ---------------------------
netcdf/4.6.3       netcdf/4.7.1           netcdf/4.7.3p  netcdf/4.8.0
netcdf/4.6.3-i8r8  netcdf/4.7.1p          netcdf/4.7.4   netcdf/4.8.0p
netcdf/4.6.3p      netcdf/4.7.3(default)  netcdf/4.7.4p

The netcdf/<version>p (henceforth referred to as 'parallel') module variants denote that these installations have been built against the parallel variants of HDF5, meaning that parallel IO using MPI-IO is enabled. Unless your application specifically uses parallel NetCDF features, e.g., by calling the parallel variants of the nc_open or nc_create functions:

int nc_open_par( const char * path, int omode, MPI_Comm comm, MPI_Info info, int* ncidp )

or

int nc_create_par( const char * path, int cmode, MPI_Comm comm, MPI_Info info, int* ncidp )

we recommend that you use the 'serial' NetCDF modules (i.e. the modules without p in the version number), even if you are compiling an MPI-enabled application.

Any NetCDF module variant ending in -i8r8 denotes that the NetCDF library has been built with default integer and float types of 8 bytes in length. If your application is not also built with 8 byte floats and integers by default, it will not build against these variants of NetCDF. We do not build these variants in the course of our standard NetCDF installations, however, they can be installed on request, if appropriate justification is provided.

The following features are enabled on all NetCDF installations on Gadi

HDF5
HDF4
Fortran Interface
NCZarr (NetCDF 4.8.0 and later)

The C++ interface is not compatible with parallel NetCDF, but has been enabled for serial NetCDF installations.

HDF5-backend compatibility

All NetCDF variants on Gadi up to 4.8.0 are built against HDF5 1.10.5 or 1.10.7. The NetCDF developers have ensured that any files created by this particular kind of NetCDF build are able to be opened, read and modified by any NetCDF installation built against HDF5 1.8. For NetCDF versions 4.7.4 and earlier, this is enforced using a restriction mechanism internal to HDF5 that prevents HDF5 1.10 creating files that cannot be read by HDF5 1.8. As of NetCDF 4.8.0, this restriction is lifted, however NetCDF does not implement any of the features listed here that would break compatibility with HDF5 1.8. Therefore, any NetCDF file created on Gadi will be able to be read by any other modern NetCDF installation, regardless of the underlying HDF5 version. Should this change in future, the details of any compatibility breaks will be added to this page.

Page tree