Rate-monotonic scheduling

Workload

Fair-share scheduling

Two-level scheduling

Scheduling (production processes)

Processor scheduling

Schedule

Mathematics

Dynamic priority scheduling

Mathematical optimization

Operating system

Parallel computing

Computer science

Algorithm

Distributed computing

Heuristics

Efficiently scheduling data processing jobs on distributed compute clusters requires complex algorithms. Current systems use simple, generalized heuristics and ignore workload characteristics, since developing and tuning a scheduling policy for each workload is infeasible. In this paper, we show that modern machine learning techniques can generate highly-efficient policies automatically.

Learning scheduling algorithms for data processing clusters