Data compression for climate data

Kuhn, M., Kunkel, J. M. and Ludwig, T. (2016) Data compression for climate data. Supercomputing Frontiers and Innovations, 3 (1). pp. 75-94. ISSN 2313-8734 doi: 10.14529/jsfi160105

Abstract/Summary

The different rates of increase for computational power and storage capabilities of supercomputers turn data storage into a technical and economical problem. Because storage capabilities are lagging behind, investments and operational costs for storage systems have increased to keep up with the supercomputers' I/O requirements. One promising approach is to reduce the amount of data that is stored. In this paper, we take a look at the impact of compression on performance and costs of high performance systems. To this end, we analyze the applicability of compression on all layers of the I/O stack, that is, main memory, network and storage. Based on the Mistral system of the German Climate Computing Center (Deutsches Klimarechenzentrum, DKRZ), we illustrate potential performance improvements and cost savings. Making use of compression on a large scale can decrease investments and operational costs by 50% without negatively impacting performance. Additionally, we present ongoing work for supporting enhanced adaptive compression in the parallel distributed file system Lustre and application-specific compression.

Item Type	Article
URI	https://reading-clone.eprints-hosting.org/id/eprint/77684
Identification Number/DOI	10.14529/jsfi160105
Refereed	Yes
Divisions	Science > School of Mathematical, Physical and Computational Sciences > Department of Computer Science
Publisher	South Urals University
Download/View statistics	View download statistics for this item