DHFS

Dynamic Hadoop Fair Scheduler (DHFS) is an optimized Hadoop Fair Scheduler that improves the performance of Hadoop by maximizing the slots utilization while guarantees the fairness across pools. It is based on the observation that at different period of time there may be idle map (or reduce) slots, as the job proceeds from map phase to reduce phase. We can use the unused map slots for those overloaded reduce tasks to improve the performance of the MapReduce workload, and vice versa, by breaking the implicit assumption that map tasks are run on map slots and reduce tasks are run on reduce slots. For example, at the beginning of MapReduce workload computation, there will be only computing map tasks and no computing reduce tasks, i.e., all the computation workload lies in the map-side. In that case, we can make use of idle reduce slots for running map tasks. Two types of DHFS are provided, namely, Pool-independent DHFS (PI-DHFS) and Pool-dependent DHFS (PD-DHFS) for users to choose.