SC20 Proceedings

The International Conference for High Performance Computing, Networking, Storage, and Analysis

Globus Services for Data-Intensive Experimental Research Automation

Workshop:Urgent HPC: HPC for Urgent Decision Making

Authors: Ryan Chard (Argonne National Laboratory (ANL))

Abstract: Lightning talk: The Globus Architecture for Data-Intensive Experimental Research (Gladier) enables the rapid development of customized automation flows linking many data services to perform data capture, storage and analysis solutions for experimental facilities. Globus provides a collection of data services built for science including: Globus Auth, Transfer, Search, Groups and Flows, and funcX to enable distributed function-as-a-service execution. Globus services are highly reliable, professionally operated cloud-hosted services that support the work of over 150,000 researchers worldwide as foundational capabilities for scientific applications and workflows: using them greatly reduces the burden on local systems, administrators and programmers.

In this lightning talk, we describe the application of these services to establish the Kanzus pipeline, integrating Argonne's Advanced Photon Source (APS) with HPC resources at the Leadership Computing Facility (ALCF) to perform online serial synchrotron crystallography (SSX) analysis to guide experimentation. The Kanzus pipeline automates capture, analysis, transfer, indexing, storage, and cataloging of tens of thousands of images for a given sample. These data are rapidly analyzed to inform the scientist about sample quality in near real-time, enabling dynamic steering of data acquisition. Once suitable data are captured the flow automatically produces quality statistics and solves the structure, reducing the time to structure from weeks to hours.


Back to Urgent HPC: HPC for Urgent Decision Making Archive Listing

Back to Full Workshop Archive Listing