Investigating read performance of Python and NetCDF when using HPC parallel filesystems

Jones, M., Blower, J., Lawrence, B. ORCID: https://orcid.org/0000-0001-9262-7860 and Osprey, A. (2016) Investigating read performance of Python and NetCDF when using HPC parallel filesystems. In: Taufer, M., Mohr, B. and Kunkel, J. M. (eds.) High Performance Computing. Lecture Notes in Computer Science, 9945. Springer International Publishing, Cham, Switzerland, pp. 153-168. ISBN 9783319460789 doi: 10.1007/978-3-319-46079-6_12

Abstract/Summary

New methods need to be developed to handle the increasing size of data sets in atmospheric science - traditional analysis scripts often inefficiently read and process the data. NetCDF4 is a common file format used in atmospheric and ocean sciences, and Python is widely used in atmospheric and ocean science data analysis. The aim of this work is to provide insight into which read patterns and sizes are most effective when using the netCDF4-python library. Quantitative information on these would be useful information for scientists, library developers, and data managers. Three different read patterns were compared to simulate different types of reads: sequential, strided, and random, with each tested across three file systems - Panasas, Lustre, and GPFS. Read rate and standard deviation were measured using Python and C, reading from plain binary files and NetCDF4 files. Read performance for netCDF4-python was compared with the performance of native Python, the C NetCDF library, and the C Posix library. As expected, comparison between the different read modes shows that access pattern and read size significantly affect achieved performance. The results also show read performance profiles that are similar for the C, C NetCDF, and Python tests, however netCDF4-python performs less efficiently.

Item Type	Book or Report Section
URI	https://reading-clone.eprints-hosting.org/id/eprint/67988
Item Type	Book or Report Section
Refereed	Yes
Divisions	Science > School of Mathematical, Physical and Computational Sciences > NCAS Science > School of Mathematical, Physical and Computational Sciences > Department of Meteorology
Publisher	Springer International Publishing
Download/View statistics	View download statistics for this item