SC20 Proceedings

The International Conference for High Performance Computing, Networking, Storage, and Analysis

Performance Tradeoffs in GPU Communication: A Study of Host and Device-Initiated Approaches


Workshop:PMBS20: The 11th International Workshop on Performance Modeling, Benchmarking and Simulation of High-Performance Computer Systems

Authors: Taylor Groves (Lawrence Berkeley National Laboratory); Ben Brock (University of California, Berkeley); Yuxin Chen (University of California, Davis); and Khaled Ibrahim, Lenny Oliker, Nicholas J. Wright, Samuel Williams, and Katherine Yelick (Lawrence Berkeley National Laboratory)


Abstract: Network communication on GPU-based systems is a significant roadblock for many applications with small but frequent messaging requirements. One common question for application developers is, 'How can they reduce the overheads and achieve the best communication performance on GPUs?' This work examines device initiated versus host initiated inter-node GPU communication using NVSHMEM. We derive basic communication model parameters for single message and batched communication before validating our model against distributed GEMM benchmarks. We use our model to estimate performance benefits for applications transitioning from CPUs to GPUS for fixed-size and scaled workloads and provide general guidelines for reducing communication overheads. Our findings show that the host-initiated approach generally outperforms the device-initiated approach for the system evaluated.





Back to PMBS20: The 11th International Workshop on Performance Modeling, Benchmarking and Simulation of High-Performance Computer Systems Archive Listing



Back to Full Workshop Archive Listing