BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/New_York
X-LIC-LOCATION:America/New_York
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20210402T160055Z
LOCATION:Track 3
DTSTART;TZID=America/New_York:20201118T103000
DTEND;TZID=America/New_York:20201118T110000
UID:submissions.supercomputing.org_SC20_sess180_pap487@linklings.com
SUMMARY:Herring: Rethinking the Parameter Server at Scale for the Cloud
DESCRIPTION:Paper\n\nHerring: Rethinking the Parameter Server at Scale for
  the Cloud\n\nThangakrishnan, Cavdar, Karakus, Ghai, Selivonchyk...\n\nTra
 ining large deep neural networks is time-consuming and may take days or ev
 en weeks to complete. Although parameter-server-based approaches were init
 ially popular in distributed training, scalability issues led the field to
  move towards all-reduce-based approaches. Recent developments in cloud ne
 tworking technologies, however, such as the Elastic Fabric Adapter (EFA) a
 nd Scalable Reliable Datagram (SRD), motivate a re-thinking of the paramet
 er-server approach to address its fundamental inefficiencies.\n\nTo this e
 nd, we introduce a novel communication library, Herring, which is designed
  to alleviate the performance bottlenecks in parameter-server-based traini
 ng. We show that gradient reduction with Herring is twice as fast as all-r
 educe-based methods. We further demonstrate that training deep learning mo
 dels like BERT using Herring outperforms all-reduce-based training, achiev
 ing 85% scaling efficiency on large clusters with up to 2048 NVIDIA V100 G
 PUs.\n\nTag: Accelerators, FPGA, and GPUs, Machine Learning, Deep Learning
  and Artificial Intelligence, Scalable Computing\n\nRegistration Category:
  Tech Program Reg Pass
END:VEVENT
END:VCALENDAR