Monitoring Hadoop by Using IEEE1888 in Implementing Energy-Aware Thread Scheduling

Hadoop is a software framework for writing applications that process huge amount of data in parallel typically on clusters of commodity computing nodes. In implementing our energy-aware scheduling framework on a Hadoop system consisting of commodity nodes, we use IEEE1888, an open network protocol, Ubiquitous Green Community Control Network Protocol. Our scheduling framework is originally developed for single-node commodity platforms and consists of a kernel level thread scheduler, API, a monitor and a controller for the behavior of the scheduler. In order to implement this framework on a Hadoop cluster, we monitor and control the slave nodes over the network by using IEEE1888. Our framework monitors performance metrics and adjusts the behavior of the scheduler through the API. The framework can effectively reduce energy consumption for memory-intensive workload on thread-enabled Hadoop.