RLScheduler: An Automated HPC Batch Job Scheduler Using Reinforcement Learning
SessionAI for IT
Event Type
Paper
File Systems and I/O
Machine Learning, Deep Learning and Artificial Intelligence
Performance/Productivity Measurement and Evaluation
Resource Management and Scheduling
TP
TimeTuesday, 17 November 20203pm - 3:30pm EDT
LocationTrack 3
DescriptionToday's high-performance computing platforms are still dominated by batch jobs. Accordingly, effective batch job scheduling is crucial to obtain high system efficiency. Existing batch job schedulers typically leverage heuristic priority functions to prioritize and schedule jobs. Once configured by the experts, such priority functions can hardly adapt to the changes of job loads, optimization goals or system settings, potentially leading to degraded system efficiency when changes occur. To address this fundamental issue, we present RLScheduler, an automated HPC batch job scheduler built on reinforcement learning. RLScheduler relies on minimal manual interventions or expert knowledge, but can learn high-quality scheduling policies via its own continuous 'trial and error'. Through extensive evaluations, we confirm that RLScheduler can learn high-quality scheduling policies towards various workloads and optimization goals with relatively low computation cost. Moreover, we show that the learned models perform stably even applied to unseen workloads, making them practical for production use.
Download PDF
