BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/New_York
X-LIC-LOCATION:America/New_York
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20210402T160101Z
LOCATION:Track 4
DTSTART;TZID=America/New_York:20201118T150000
DTEND;TZID=America/New_York:20201118T153000
UID:submissions.supercomputing.org_SC20_sess177_pap493@linklings.com
SUMMARY:Reducing Communication in Graph Neural Network Training
DESCRIPTION:Paper\n\nReducing Communication in Graph Neural Network Traini
ng\n\nTripathy, Yelick, Buluc\n\nGraph Neural Networks (GNNs) are powerful
and flexible neural networks that use the naturally sparse connectivity i
nformation of the data. GNNs represent this connectivity as sparse matrice
s, which have lower arithmetic intensity and thus higher communication cos
ts compared to dense matrices, making GNNs harder to scale to high concurr
encies than convolutional or fully-connected neural networks.\n\nWe introd
uce a family of parallel algorithms for training GNNs and show that they c
an asymptotically reduce communication compared to previous parallel GNN t
raining methods. We implement these algorithms, which are based on 1D, 1.5
D, 2D and 3D sparse-dense matrix multiplication, using torch.distributed o
n GPU-equipped clusters. Our algorithms optimize communication across the
full GNN training pipeline. We train GNNs on over a hundred GPUs on multip
le datasets, including a protein network with over a billion edges.\n\nTag
: Accelerators, FPGA, and GPUs, Linear Algebra, Machine Learning, Deep Lea
rning and Artificial Intelligence\n\nRegistration Category: Tech Program R
eg Pass
END:VEVENT
END:VCALENDAR