Taming I/O Variation on QoS-Less HPC Storage: What Can Applications Do?

Authors: Zhenbo Qiao (New Jersey Institute of Technology); Qing Liu (New Jersey Institute of Technology, Oak Ridge National Laboratory (ORNL)); and Norbert Podhorszki, Scott Klasky, and Jieyang Chen (Oak Ridge National Laboratory (ORNL))

Abstract: As high-performance computing (HPC) is being scaled up to exascale to accommodate new modeling and simulation needs, I/O has continued to be a major bottleneck in end-to-end scientific processes. This work aims to take advantage of the storage characteristics and explore application level solutions that are interference-aware. In particular, we monitor the performance of data analytics and estimate the state of shared storage resources using discrete fourier transform. If there is heavy I/O interference, data analytics can dynamically adapt to the environment by lowering the accuracy and performing partial or no augmentation from the shared storage, dictated by an augmentation-bandwidth plot. We evaluate three data analytics; XGC, GenASiS and Jet; on Chameleon, and quantitatively demonstrate that both the average and variation of I/O performance can be vastly improved, with the mean and variance improved by as much as 18x and 60x, respectively, while maintaining acceptable outcome of data analysis.




Back to Technical Papers Archive Listing