ROBIN (RuciO/BIgData Express/SENSE): A Next-Generation High-Performance Data Service Platform
Data Analytics, Compression, and Management
Performance/Productivity Measurement and Evaluation
TimeThursday, 12 November 20203:40pm - 4:05pm EDT
DescriptionIncreasingly, scientific research discovery depends on the analysis of extremely large amounts of data. This data is gathered through very large scientific instruments, very large numbers of small devices, and mid-scale instruments. These devices generate exponentially increasing volumes of data. Such data is often obtained, analyzed, stored, visualized, and transferred at and among sites world-wide by large collaborations of science communities. Managing and moving extremely large volumes of data across today’s networks is a special multidimensional challenge. Also, existing data management and movement services and tools often are inadequate to address all specific requirements. This paper describes the various components of this challenge and how those issues can be addressed by ROBIN, a unique comprehensive set of integrated services designed specifically for managing and moving extremely large amounts of data over long distances, e.g., thousands of miles around the globe. The services include Rucio, a data management service widely used by particle physics research communities, BigData Express, a schedulable, predictable, and high-performance data transfer service, and SENSE, a distributed network resource orchestrator. This paper also describes the results of initial experiments using that set of integrated services.