SC20 Proceedings

The International Conference for High Performance Computing, Networking, Storage, and Analysis

Pangeo Benchmarking Analysis: Object Storage vs. POSIX File System

Workshop:Fifth International Parallel Data Systems Workshop

Authors: Haiying Xu, Kevin Paul, and Anderson Banihirwe (National Center for Atmospheric Research (NCAR))

Abstract: Pangeo is a community of scientists and software developers collaborating to enable Big Data Geoscience analysis interactively in the public cloud and on high-performance computing (HPC) systems. At the core of the Pangeo software stack is (1) Xarray, which adds labels to metadata such as dimensions, coordinates and attributes for raw array-oriented data, (2) Dask, which provides parallel computation and out-of-core memory capabilities, and (3) Jupyter Lab which offers the web-based interactive environment to the Pangeo platform.

Geoscientists now have a strong candidate software stack to analyze large datasets, and they are very curious about performance differences between the Zarr and NetCDF4 data formats on both traditional file storage systems and object storage. We have written a benchmarking suite for the Pangeo stack that can measure scalability and performance information of both input/output (I/O) throughput and computation. We will describe how we performed these benchmarks, analyzed our results, and we will discuss the pros and cons of the Pangeo software stack in terms of I/O scalability on both cloud and HPC storage systems.


Back to Fifth International Parallel Data Systems Workshop Archive Listing

Back to Full Workshop Archive Listing