The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop:INDIS 2020: The 7th International Workshop on Innovating the Network for Data-Intensive Science
Authors: Wenji Wu, Liang Zhang, Qiming Lu, Phil Demar, and Robert Illingworth (Fermi National Accelerator Laboratory); Joe Mambretti, Se-Young Yu, and Jim Hao Chen (Northwestern University); and Inder Monga, Xi Yang, Tom Lehman, Chin Guok, and John Macauley (Energy Sciences Network (ESnet))
Abstract: Increasingly, scientific research discovery depends on the analysis of extremely large amounts of data. This data is gathered through very large scientific instruments, very large numbers of small devices, and mid-scale instruments. These devices generate exponentially increasing volumes of data. Such data is often obtained, analyzed, stored, visualized, and transferred at and among sites world-wide by large collaborations of science communities. Managing and moving extremely large volumes of data across today’s networks is a special multidimensional challenge. Also, existing data management and movement services and tools often are inadequate to address all specific requirements. This paper describes the various components of this challenge and how those issues can be addressed by ROBIN, a unique comprehensive set of integrated services designed specifically for managing and moving extremely large amounts of data over long distances, e.g., thousands of miles around the globe. The services include Rucio, a data management service widely used by particle physics research communities, BigData Express, a schedulable, predictable, and high-performance data transfer service, and SENSE, a distributed network resource orchestrator. This paper also describes the results of initial experiments using that set of integrated services.