SC20 Proceedings

The International Conference for High Performance Computing, Networking, Storage, and Analysis

Toward an Automated HPC Pipeline for Processing Large Scale Electron Microscopy Data

Workshop:XLOOP 2020: 2nd Annual Workshop on Extreme-Scale Experiment-in-the-Loop-Computing

Authors: Rafael Vescovi (Argonne National Laboratory (ANL)), Hanyu Li (University of Chicago), Jeffery Kinnison (University of Notre Dame), Murat Keceli and Misha Salim (Argonne National Laboratory (ANL)), Narayanan Kasthuri (University of Chicago), and Thomas D. Uram and Nicola Ferrier (Argonne National Laboratory (ANL))

Abstract: We present a fully modular and scalable software pipeline for processing electron microscope (EM) images of brain slices into 3D visualization of individual neurons and demonstrate an end-to-end segmentation of a large EM volume using a supercomputer. Our pipeline scales multiple packages used by the EM community with minimal changes to the original source codes. We tested each step of the pipeline individually, on a workstation, a cluster and a supercomputer. Furthermore, we can compose workflows from these operations using a Balsam database that can be triggered during the data acquisition or with the use of different front ends, and control the granularity of the pipeline execution. We describe the implementation of our pipeline and modifications required to integrate and scale up existing codes. The modular nature of our environment enables diverse research groups to contribute to the pipeline without disrupting the workflow, i.e., new individual codes can be easily integrated for each step on the pipeline.

Back to XLOOP 2020: 2nd Annual Workshop on Extreme-Scale Experiment-in-the-Loop-Computing Archive Listing

Back to Full Workshop Archive Listing