A novel parallel hybrid PSO-GA using MapReduce to schedule jobs in Hadoop data grids

Scheduling heterogeneous tasks in a heterogeneous grid environment aims at effectively utilizing the resources and sharing the load among the available resources. Such a task assignment problem is NP-hard. This paper presents a Hybrid Particle Swarm Optimization - Genetic Algorithm (HPSO-GA) for solving the Task Assignment Problem. The novel Particle Swarm Optimization (PSO) implements GA operations such as crossover and mutation in PSO to improve effective resource utilization and complete tasks within deadline. The algorithm aims at distributing load among the heterogeneous resources in the grid environment based on their capacity. Analysis of data and computation intensive applications like web log processing and bioinformatics to achieve optimal performance is time consuming. Hence parallelization of optimization function is essential. Large-scale parallellisation of optimization function must also guarantee efficient communication, load balancing, fault tolerance and reliability. This paper presents a MapReduce HPSO-GA based on MapReduce parallel programming model. The HPSO-GA yields better results than normal PSO, provides better load balancing and resource utilization in grid environment. It identifies the exact node to which a task can be assigned in a Hadoop cluster. Hence, the proposed approach can be used in the resource management system of Hadoop along with Hadoop and system parameters to schedule jobs efficiently in a Hadoop cluster.