<abstract xmlns="http://www.w3.org/1999/xhtml">

<p>A lot of scientific work is dedicated to the analysis of data. Most of the analyzed data, like data from space missions, are structured. The choice of data format can affect various characteristics - read/write speed of standard files, read/write speed of small files and read/write speed of compressed data formats. In this paper, we analyze binary data formats, proposed types of the tests and testing methods, and compare their performance with human-readable text format. We also discuss compressed and uncompressed modes available for data formats like HDF5 and netCDF. When disregarding precision, the best data format from the size perspective is lossy HDF5 without compression. Losless HDF5 without compression show the best speed performance. Lossy HDF5 without compression is the best balance between size reduction and speed. However, for specific criteria and types of files, there might be better candidates as detailed in the conclusion.</p>
</abstract>

A lot of scientific work is dedicated to the analysis of data. Most of the analyzed data, like data from space missions, are structured. The choice of data format can affect various characteristics - read/write speed of standard files, read/write speed of small files and read/write speed of compressed data formats. In this paper, we analyze binary data formats, proposed types of the tests and testing methods, and compare their performance with human-readable text format. We also discuss compressed and uncompressed modes available for data formats like HDF5 and netCDF. When disregarding precision, the best data format from the size perspective is lossy HDF5 without compression. Losless HDF5 without compression show the best speed performance. Lossy HDF5 without compression is the best balance between size reduction and speed. However, for specific criteria and types of files, there might be better candidates as detailed in the conclusion.

Which Data Format To Store Scientific Data Should I Use? A Performance Analysis

Department of Computers and Informatics, Faculty of Electrical Engineering and Informatics

Acta Electrotechnica et Informatica

This work is licensed under the Creative Commons Attribution 4.0 International License.

{"article-title":"Which Data Format To Store Scientific Data Should I Use? A Performance Analysis"}

A lot of scientific work is dedicated to the analysis of data. Most of the analyzed data, like data from space missions, are structured. The choice of data...

Which Data Format To Store Scientific Data Should I Use? A Performance Analysis

Published Online: Oct 14, 2022

Page range: 32 - 40

Received: May 01, 2022

Accepted: Jun 13, 2022

DOI: https://doi.org/10.2478/aei-2022-0015

Keywords
scientific data, data storage, compression, performance

© 2022 Daniel Gecášek et al., published by Sciendo

This work is licensed under the Creative Commons Attribution 4.0 International License.

Which Data Format To Store Scientific Data Should I Use? A Performance Analysis

Published Online: Oct 14, 2022

Page range: 32 - 40

Received: May 01, 2022

Accepted: Jun 13, 2022

DOI: https://doi.org/10.2478/aei-2022-0015

Keywordsscientific data, data storage, compression, performance

© 2022 Daniel Gecášek et al., published by Sciendo

This work is licensed under the Creative Commons Attribution 4.0 International License.

Keywords
scientific data, data storage, compression, performance