Rapid Processing of Astronomical Data for the Dark Energy Spectroscopic Instrument
Event Type
Workshop
Applications
Scientific Computing
Simulation
State of the Practice
Technology Challenge
W
TimeFriday, 13 November 20203:25pm - 3:50pm EDT
LocationTrack 10
DescriptionDark energy remains one of the least understood areas of modern cosmology and physics. The Dark Energy Spectroscopic Instrument (DESI) was built to improve our understanding through the acquisition of data from over thirty million galaxies. DESI is a highly multiplexed instrument capable of acquiring astronomical spectra for 5000 objects simultaneously. This multiplexing, which is larger than any multi-object spectrograph to date, is made more ambitious with an observing strategy that requires up-to-date information of data quality and completeness on a daily basis to inform the scheduling of future observations. With an order of magnitude larger data volume than its predecessors and more stringent survey demands, novel methods have been developed to rapidly process the data in near-real-time and make inferences based on the results. The data acquired from thirty highly sensitive cameras are transferred from local storage on the mountaintop to the National Energy Research Scientific Computing Center (NERSC). Using an urgent queue operating at NERSC, a parallelized set of routines processes the images and extracts astronomically relevant information within minutes to hours of acquisition, enabling daytime analysis by the operations manager to make observing program decisions for the following night. The implementation has been run at scale and tested in the spring of 2020 during commissioning of the instrument and proven successful at giving critical feedback to the observing team about the previous night's data quality. Quantitative and qualitative results will be discussed, along with planned improvements to the workflow before the survey begins full operation in 2021.