SC20 Is Everywhere We Are

SC20 Virtual Platform
Pangeo Benchmarking Analysis: Object Storage vs. POSIX File System
Event Type
Big Data
Data Analytics, Compression, and Management
Data Movement
File Systems and I/O
Registration Categories
TimeThursday, 12 November 20202:57pm - 3:23pm EST
LocationTrack 5
DescriptionPangeo is a community of scientists and software developers collaborating to enable Big Data Geoscience analysis interactively in the public cloud and on high-performance computing (HPC) systems. At the core of the Pangeo software stack is (1) Xarray, which adds labels to metadata such as dimensions, coordinates and attributes for raw array-oriented data, (2) Dask, which provides parallel computation and out-of-core memory capabilities, and (3) Jupyter Lab which offers the web-based interactive environment to the Pangeo platform.

Geoscientists now have a strong candidate software stack to analyze large datasets, and they are very curious about performance differences between the Zarr and NetCDF4 data formats on both traditional file storage systems and object storage. We have written a benchmarking suite for the Pangeo stack that can measure scalability and performance information of both input/output (I/O) throughput and computation. We will describe how we performed these benchmarks, analyzed our results, and we will discuss the pros and cons of the Pangeo software stack in terms of I/O scalability on both cloud and HPC storage systems.
Back To Top Button