Geophysics Community

Page tree

Time series metadata

  1. mt_metadata: In 2019, the Working Group for Magnetotelluric Data Handling and Software (https://www.iris.edu/hq/about_iris/governance/mt_soft) was assembled by the Incorporated Research Institutions for Seismology (IRIS) to develop a metadata standard for MT time series data. The metadata definitions for the mt_metadata standard are described in Peacock et al., 2021. Additionally, the open-source Python package mt_metadata, which can be used to read/write/manipulate magnetotelluric metadata, is described in Peacock et al., 2022.

  2. EMERALD: The acronym EMERALD was supposed to stand for ElectroMagnetic Equipment, Raw data And Locations Database. What survived over the years was the EMERALD processing, a set of computer programs to analyse MT time series data, and the EMERALD file format for storing MT data. EMERALD data files typically come in pairs of two files with the same name but differing file name extensions, sometimes called RAW and XTR files. XTR (extract) files are plain ASCII files, which can be read and modified with text editors. RAW files or more generally, EMERALD-type data files are in most cases binary and used to store all kind of magnetotelluric (MT) data such as time series, cross- and auto spectra and calibration data. The EMERALD-type data files store any number of channels of data in matrix form, but do not contain any description of the data. This information is stored in the according XTR file. In 2015 the original XTR files were replaced by a modernized version based on the Extensible Markup Language (XML). The new files have the extension .XTRX. More information on the EMERALD Data Format can be found at https://doi.org/10.2312/GFZ.b103-15082.

  3. Metadata for field acquisition: The Magnetotelluric Time Series Metadata Consultation and Recommendations (see Leonard et al., 2021 and  Duan et al., 2021) focused on the definitions of metadata that should be collected along with the raw data of an MT survey.

Time series data formats

  1. MTH5 is an HDF5 data container for magnetotelluric time series data, but could be extended to other data types. MTH5 uses h5py to interact with the HDF5 file, xarray to interact with the data in a nice way, and all metadata use mt_metadata. Please see the mth5 documentation for examples on how to use and work with MTH5 files.

  2. EMERALD: A set of computer programs to analyse MT time series data, and the EMERALD file format for storing MT data. More information on the EMERALD Data Format can be found at https://doi.org/10.2312/GFZ.b103-15082

  3. NetCDF-4/HDF5 is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. Some example published NetCDF-4/HDF5 MT time series data can be found at https://dx.doi.org/10.25914/5eaa30de53244.

  4. TileDB Embedded: TileDB Embedded has been developed by TileBD Inc. and is an open-source storage engine architected around dense and sparse multi-dimensional arrays. TileDB Embedded enables storing and accessing Dense arrays (e.g., images, video and more), Sparse arrays (e.g., LiDAR, genomics and more), Dataframes (any tabular data, as either dense or sparse arrays) and any data that can be modeled as arrays (e.g., graphs, key-values, ML models, etc.). Time-series data can be modeled by a 2D array, either dense with labeled dimensions or sparse with datetime and string dimensions. The TileDB Embedded storage format is particularly optimized for storing and retrieving data on cloud object stores such as AWS S3, Azure Blob Storage and Google Cloud Storage, as well as on any other object store, such as MinIO. All data writes handled by TileDB Embedded are immutable and timestamped allowing for the ability to time travel and the absolute control over writes and reads on arrays for maximum reproducibility.

    The Data Services of IRIS and the Geodetic Data Services of UNAVCO are designing, developing and implementing a cloud-based platform that will provide services for data queries across their internal repositories allowing researchers  to conduct their data processing in the same, or data-proximate, cloud as the platform (Tradbant et al., 2022). This cloud-based platform will adopt cloud-optimized data containers such as TileDB Embedded, which will allow for more efficient processing.

  5. American Standard Code for Information Interchange (US-ASCII) / text files.

  6. ts_format: Time series format for input to LIMS processing codes. Each time series file contains three sections. A comment block, an information block, and a data block. The file can either be in plain ASCII, or in formatted or unformatted BINARY. The channel order used is: Hx, Hy, Hz, Ex, Ey. For more information, please refer to the ts_format documentation.

Transfer function metadata

  1. EMTF XML: A set of minimal metadata fields are defined to make the MT transfer function data self-descriptive. These include formal copyright and conditions of use, geographic name and location, data quality identifiers, data provenance, processing assumptions, and time period and period range. 

Transfer function data formats

  1. EMTF XML: Electromagnetic transfer function extensible markup language (EMTF XML) is a novel, self-describing, searchable, and extensible way to store transfer function data and metadata. For more information, please see Kelbert, A., 2020.

  2. EDI: The Society of Exploration Geophysicists (SEG) Data Interchange Standard (Wight, 1987), also known as the Electrical Data Interchange (EDI), is a standard file for the interchange of data from magnetotelluric (MT), electromagnetic array profiling (EMAP) or similar electrical geophysical techniques.

  3. JavaScript Object Notation (JSON): JSON is a language independent standard text-based format for representing structured data based on the JavaScript object syntax and is commonly used in electronic data interchange. An example of JSON MT TF's can be found here: http://mt.research.ltu.se/MT/POLAND/EPOS-PL.GRD__.2014.tf.json  

  4. Magnetotelluric data file J-format: A file of MT data responses consisting of a comment block, followed by an information block, followed by one or more data blocks for that site. See J-format documentation and J-format example file.

  5. EMTF Z-files (Eisel and Egbert, 2001)

  6. BIRRP format: see Chave et al., 1987 and the BIRRP user guide.

References

Chave, A.D., Thomson, D.J. and Ander, M.E., 1987. On the robust estimation of power spectra, coherences, and transfer functions. Journal of Geophysical Research: Solid Earth, 92(B1), pp.633-648. https://doi.org/10.1029/JB092iB01p00633

Duan, J., Kirkby, A., Kyi, D., Jiang, W., Costelloe, M. and Hitchman, A., 2021. Metadata standards for magnetotelluric time-series data. Preview, 2021(215), pp.61-63. https://doi.org/10.1080/14432471.2021.2012035

Eisel, M. and Egbert, G.D., 2001. On the stability of magnetotelluric transfer function estimates and the reliability of their variances. Geophysical Journal International, 144(1), pp.65-82. https://doi.org/10.1046/j.1365-246x.2001.00292.x

Kelbert, A., 2020, EMTF XML: New data interchange format and conversion tools for electromagnetic transfer functions, Geophysics, 85: F1-F17. https://doi.org/10.1190/geo2018-0679.1

Leonard, T., Fitzgerald, D., Keetley, J., 2021. Magnetotelluric Time Series Metadata Consultation and Recommendations. Geoscience Australia, Canberra. http://pid.geoscience.gov.au/dataset/ga/145467

Peacock, J.R., Frassetto, A., Kelbert, A., Egbert, G., Smirnov, M., Schultz, A.C., Kappler, K.N., Ronan, T., and Trabant, C., 2021, Metadata Standards for Magnetotelluric Time Series Data: U.S. Geological Survey data release, https://doi.org/10.5066/P9AXGKEV.

Peacock, J., Kappler, K., Heagy, L., Ronan, T., Kelbert, A. and Frassetto, A., 2022. MTH5: An archive and exchangeable data format for magnetotelluric time series data. Computers & Geosciences, 162, p.105102. https://doi.org/10.1016/j.cageo.2022.105102

Trabant, C., Berglund, H., Carter, J., and Mencin, D.: Developing a Next Generation Platform for Geodetic, Seismological and Other Geophysical Data Sets and Services, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-8905, https://doi.org/10.5194/egusphere-egu22-8905, 2022.

Wight, D.E., 1987. Society of Exploration Geophysicists MT/EMAP Data Interchange Standard.


  • No labels