HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of datatypes, and is designed for flexible and efficient I/O and for high volume and complex data. HDF5 is portable and is extensible, allowing applications to evolve in their use of HDF5. The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format. For more information on using the HDF5 library, or any of the provided utilities see the HDF5 support page at the HDFgroup website.
To use HDF5 on Gadi, first load one of the HDF5 modules.
$ module load hdf5/1.10.7 |
Please note that it is important to specify a version when loading a HDF5 module, as some files created by later HDF5 versions cannot be opened by earlier versions. In order to build an application on Gadi against HDF5, only the required HDF5 libraries need to be linked
$ gcc -o read_h5_file read_h5_file.c -lhdf5 -lhdf5_hl |
The libraries that HDF5 depends on are linked automatically by default using data stored in the shared HDF5 libraries. You do not need to load any underlying modules in order to compile your application. To use the fortran interface, the fortran libraries also need to be linked
$ gfortran -o read_h5_file read_h5_file.f90 -lhdf5_fortran -lhdf5hl_fortran -lhdf5 -lhdf5_hl |
Every HDF5 installation on Gadi has been built to be compatible with all GNU and Intel compilers available on Gadi. In the above commands, icc can be substituted for gcc and ifort can be substituted for gfortran without any other changes. The HDF5 libraries corresponding to your choice of compiler are selected at compile time by NCI's compiler wrappers. Every parallel HDF5 installation on Gadi is compatible with every MPI distribution installed on Gadi by a similar mechanism, with one minor exception:
OMPI_MCA_io=ompio or pass --mca io ompio to mpirun when you run your application. This is due to a break in compatibility between the default MPI-IO implementation between OpenMPI 3 and 4, which are otherwise binary compatible.We do not usually recommend the use of static linking on Gadi, as it prevents applications from receiving important security or bug fixes that would otherwise be applied automatically when shared libraries are updated. If your application uses HDF5, and is compiled statically (e.g. by passing the -static flag to the compiler) then you will need to include the libraries that HDF5 depends on. The easiest way to do this is to use the compiler wrappers provided with HDF5
$ h5cc -static -o read_h5_file read_h5_file.c $ h5fc -static -o read_h5_file read_h5_file.f90 |
These wrappers can be used with both the Intel and GNU compilers. The underlying compiler used by each wrapper is selected based on the modules loaded in your environment. Note that MPI applications cannot be built statically on Gadi.
As of HDF5 1.12, releases are now divided into 'Experimental' and 'Maintenance' releases, as outlined in the Introducing Experimental Releases page. For this reason, NCI will not install odd-numbered minor (i.e. 'Experimental') releases unless a specific feature new to that release is requested by users. NCI will continue to proactively support even-numbered minor (i.e. 'Maintenance') releases. These will be installed on /apps as soon as practical after they become available. As is standard policy for /apps, NCI will not install any versions of HDF5 that were released earlier than when Gadi became generally available to users.
There are two HDF5 modules available for each software version:
$ module avail hdf5 -------------------------- /apps/Modules/modulefiles --------------------------- hdf5/1.8.21 hdf5/1.10.5 hdf5/1.10.7(default) hdf5/1.12.1 hdf5/1.8.21p hdf5/1.10.5p hdf5/1.10.7p hdf5/1.12.1p |
The hdf5/<version>p (henceforth referred to as 'parallel') module variants denote that these installations have been built with parallel IO enabled. Unless your application specifically uses parallel HDF5 features, e.g., by invoking the MPI-IO VFD using
herr_t H5Pset_fapl_mpio( hid_t fapl_id, MPI_Comm comm, MPI_Info info ) |
we recommend that you use the serial HDF5 modules (i.e. the modules without p in the version number), even if you are compiling an MPI-enabled application.
The following features are enabled for all HDF5 installations on Gadi:
The C++ interface is not compatible with parallel HDF5, but has been enabled on the serial HDF5 installations.
With the release of HDF5 1.10, new features were introduced that, when enabled for a particular file, cause that file to be unreadable to HDF5 1.8 and earlier. Specifically, creating a file with any of the following features enabled:
Or creating a file with H5P_SET_LIBVER_BOUNDS ( fapl_id, low, high ) where low=H5F_LIBVER_V110 will cause that file to be unreadable by HDF5 1.8. See The HDF5 documentation for new features in HDF5 1.10 for more information. If you are concerned that the HDF5 files your application creates will need to be opened on another machine that does not have an HDF5 1.10 installation or later, or by old software that cannot use HDF5 1.10, we advise that you build your application against hdf5/1.8.21 . Applications that build against HDF5 1.8 should be able to be built against HDF5 1.10 with no modification to the source.
HDF5 1.12 introduces further new features, and as a result, there is an API break between HDF5 1.10 and HDF5 1.12. Applications that build against HDF5 1.10 cannot be built against HDF5 1.12 without modification. Though there is a compatibility option provided to build HDF5 1.12 with a 1.10 compatible API, we have not enabled this feature on Gadi as we have several HDF5 1.10 builds, and using the HDF5 1.10 API disables use of new HDF5 1.12 features. For more information see the Migrating From HDF5 1.10 To HDF5 1.12 on the HDFgroup website.