SC20 Proceedings

The International Conference for High Performance Computing, Networking, Storage, and Analysis

Petascale XCT: 3D Image Reconstruction with Hierarchical Communications on Multi-GPU Nodes

Authors: Mert Hidayetoglu (University of Illinois); Tekin Bicer (Argonne National Laboratory); Simon Garcia de Gonzalo (Barcelona Supercomputing Center); Bin Ren (College of William & Mary); Vincent De Andrade, Doga Gursoy, Rajkumar Kettimuthu, and Ian T. Foster (Argonne National Laboratory); and Wen-mei W. Hwu (University of Illinois)

Abstract: X-ray computed tomography is a commonly used technique for noninvasive imaging at synchrotron facilities. Iterative tomographic reconstruction algorithms are often preferred for recovering high quality 3D volumetric images from 2D X-ray images; their use, however, has been limited to small/medium datasets due to their computational requirements. In this paper, we propose a high-performance iterative reconstruction system for terabyte(s)-scale 3D volumes. Our design involves three novel optimizations: (1) optimization of (back)projection operators by extending the 2D memory-centric approach to 3D; (2) inclusion of hierarchical communications by exploiting “fat-node” architecture with many GPUs; (3) utilization of mixed-precision types while preserving convergence rate and quality. We extensively evaluate the proposed optimizations and scaling on the Summit supercomputer. Our largest reconstruction is a mouse brain volume with 9K×11K×11K voxels, where the total reconstruction time is under three minutes using 24,576 GPUs, reaching 65 PFLOPS; 34% of Summit's peak performance.

Back to Technical Papers Archive Listing