SciStream: Architecture and Toolkit for Data Streaming between Federated Science Instruments
Presenter
Event Type
Workshop
Applications
Scientific Computing
Simulation
State of the Practice
Technology Challenge
W
TimeFriday, 13 November 20205:45pm - 5:50pm EDT
LocationTrack 10
DescriptionLightning talk: Scientists often wait in long queues to perform experiments, observations and simulations on advanced scientific instruments. It may take hours to days to get access to supercomputers, months to get time on light sources and days to months or even longer to observe phenomena of interest on telescopes. To utilize these resources efficiently, researchers often need to analyze data streaming from scientific instruments in near-real time, so that results from one experiment (or simulation) can guide selection of the next or even influence the course of a single experiment (or simulation). Such analyses allow researchers to identify and correct problems in the experimental setup almost immediately. Realization of such near-real-time analysis requires effective and reliable methods for acquisition of network and remote HPC resources at a specific time for a specific period, efficient and secure data streaming from scientific instruments to remote compute nodes, and analysis of data streams at data generation rates so that timely decisions can be taken.
Our SciStream project focuses on developing architecture and protocols for efficient and secure data streaming from the data producer’s memory to the data consumer’s memory through seamless integration of networking protocols and technologies with science application layer processes. In this talk, we will briefly describe the SciStream architecture and a working prototype that was used in an award-winning live demonstration at SCinet Technology Challenge in SC19.
Our SciStream project focuses on developing architecture and protocols for efficient and secure data streaming from the data producer’s memory to the data consumer’s memory through seamless integration of networking protocols and technologies with science application layer processes. In this talk, we will briefly describe the SciStream architecture and a working prototype that was used in an award-winning live demonstration at SCinet Technology Challenge in SC19.
