Adaptive Pre-emption-Aware Hierarchical Scheduling with Recovery Queues for Hadoop YARN

Abstract

Efficient job scheduling remains a fundamental challenge in Hadoop YARN because dynamic workloads, frequent resource contention, and task pre-emption often lead to increased waiting time, redundant execution, and reduced resource utilization. Conventional scheduling policies primarily focus on resource allocation and job ordering but provide limited support for preserving execution progress after task interruption. This paper presents an Adaptive Pre-emption-Aware Hierarchical Scheduling with Recovery Queues (APHRQ) framework to improve scheduling efficiency through adaptive decision-making and recovery-aware execution. The proposed framework combines adaptive score-based job selection, controlled pre-emption, checkpoint-assisted recovery, a dedicated Recovery Queue, and a Protected Queue to enable interrupted jobs to resume execution from their previously saved state while preventing repeated interruptions. A Recovery-Aware Fairness Index (RAFI) is also introduced to evaluate fairness by considering the scheduling opportunities afforded to interrupted workloads, in addition to conventional resource-allocation fairness. The proposed framework was evaluated using a Python-based Hadoop YARN simulation developed with Streamlit under normal, pre-emption-heavy, and critical workload scenarios. Comparative evaluation against FIFO, Fair Scheduler, Capacity Scheduler, Shortest Job First (SJF), Shortest Remaining Time First (SRTF), and a pre-emption-based scheduler without recovery support demonstrates that APHRQ improves throughput, resource utilization, recovery efficiency, and recovery-aware fairness while reducing redundant re-computation and starvation under dynamic workload conditions. The results indicate that integrating adaptive scheduling with checkpoint-assisted recovery provides a balanced and efficient scheduling strategy for Hadoop YARN environments.