BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/New_York
X-LIC-LOCATION:America/New_York
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20210402T160557Z
LOCATION:Track 7
DTSTART;TZID=America/New_York:20201111T153000
DTEND;TZID=America/New_York:20201111T160000
UID:submissions.supercomputing.org_SC20_sess199_ws_dls106@linklings.com
SUMMARY:DDLBench: Towards a Scalable Benchmarking Infrastructure for Distr
 ibuted Deep Learning
DESCRIPTION:Workshop\n\nDDLBench: Towards a Scalable Benchmarking Infrastr
 ucture for Distributed Deep Learning\n\nJansen, Codreanu, Varbanescu\n\nDu
 e to its many applications across various fields of research, engineering 
 and daily life, deep learning has seen a surge in popularity. Therefore, l
 arger and more expressive models have been proposed, with examples like Tu
 ring-NLG using as many as 17 billion parameters. Training these very large
  models becomes increasingly difficult due to the high computational costs
  and large memory footprint. Therefore, several approaches for distributed
  training based on data parallelism (e.g., Horovod) and model/pipeline par
 allelism (e.g., GPipe, PipeDream) have emerged. In this work, we focus on 
 an in-depth comparison of three different parallelism models that address 
 these needs: data, model and pipeline parallelism. To this end, we provide
  an analytical comparison of the three, both in terms of computation time 
 and memory usage, and introduce DDLBench, a comprehensive (open-source, re
 ady-to-use) benchmark suite to quantify these differences in practice. Thr
 ough in-depth performance analysis and experimentation with various models
 , datasets, distribution models and hardware systems, we demonstrate that 
 DDLBench can accurately quantify the capability of a given system to perfo
 rm distributed deep learning (DDL).\n\nBy comparing our analytical models 
 with the benchmarking results, we show how the performance of real-life im
 plementations diverges from these analytical models, thus requiring benchm
 arking to capture the in-depth complexity of the frameworks themselves.\n\
 nRegistration Category: Workshop Reg Pass
END:VEVENT
END:VCALENDAR