SC20 Proceedings

The International Conference for High Performance Computing, Networking, Storage, and Analysis

Rapid Processing of Astronomical Data for the Dark Energy Spectroscopic Instrument


Workshop:Urgent HPC: HPC for Urgent Decision Making

Authors: Anthony Kremin and Stephen Bailey (Lawrence Berkeley National Laboratory)


Abstract: Dark energy remains one of the least understood areas of modern cosmology and physics. The Dark Energy Spectroscopic Instrument (DESI) was built to improve our understanding through the acquisition of data from over thirty million galaxies. DESI is a highly multiplexed instrument capable of acquiring astronomical spectra for 5000 objects simultaneously. This multiplexing, which is larger than any multi-object spectrograph to date, is made more ambitious with an observing strategy that requires up-to-date information of data quality and completeness on a daily basis to inform the scheduling of future observations. With an order of magnitude larger data volume than its predecessors and more stringent survey demands, novel methods have been developed to rapidly process the data in near-real-time and make inferences based on the results. The data acquired from thirty highly sensitive cameras are transferred from local storage on the mountaintop to the National Energy Research Scientific Computing Center (NERSC). Using an urgent queue operating at NERSC, a parallelized set of routines processes the images and extracts astronomically relevant information within minutes to hours of acquisition, enabling daytime analysis by the operations manager to make observing program decisions for the following night. The implementation has been run at scale and tested in the spring of 2020 during commissioning of the instrument and proven successful at giving critical feedback to the observing team about the previous night's data quality. Quantitative and qualitative results will be discussed, along with planned improvements to the workflow before the survey begins full operation in 2021.





Back to Urgent HPC: HPC for Urgent Decision Making Archive Listing



Back to Full Workshop Archive Listing