FLEX: A Slot Allocation Scheduling Optimizer for MapReduce Workloads
Abstract
Originally, MapReduce implementations such as Hadoop
employed First In First Out (fifo) scheduling, but such simple schemes
cause job starvation. The Hadoop Fair Scheduler (hfs) is a slot-based
MapReduce scheme designed to ensure a degree of fairness among the jobs,
by guaranteeing each job at least some minimum number of allocated
slots. Our prime contribution in this paper is a different, flexible
scheduling allocation scheme, known as flex. Our goal is to optimize any
of a variety of standard scheduling theory metrics (response time,
stretch, makespan and Service Level Agreements (slas), among others)
while ensuring the same minimum job slot guarantees as in hfs, and
maximum job slot guarantees as well. The flex allocation scheduler can
be regarded as an add-on module that works synergistically with hfs. We
describe the mathematical basis for flex, and compare it with fifo and
hfs in a variety of experiments.
Domains
Digital Libraries [cs.DL]Origin | Files produced by the author(s) |
---|
Loading...