The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop:XLOOP 2020: 2nd Annual Workshop on Extreme-Scale Experiment-in-the-Loop-Computing
Authors: Dilworth Parkinson, Harinarayan Krishnan, Daniela Ushizima, Matthew Henderson, and Shreyas Cholia (Lawrence Berkeley National Laboratory)
Abstract: The constant stream of new users, samples and experimental approaches at many light source beam lines means a single refined set of processing steps and parameters can seldom be re-used, and instead an interactive approach to exploring parameter space is required. But the many large data sets often generated during experiments mean that testing even one set of parameters is time-intensive. We present two approaches to leverage parallel high-performance computing to quickly and interactively explore parameter space for a large number of data sets, in the context of synchrotron tomography. The first approach uses a combination of template Jupyter notebooks with custom widgets, along with Papermill and Dask. The second approach leverages the workflow infrastructure of Xi-Cam and Cam-Link, an ecosystem designed for real-time interactive exploration of data at user facilities.