Title :
CoHadoop++: A load balanced data co-location in Hadoop Distributed File System
Author :
Nishanth, S. ; Radhikaa, B. ; Ragavendar, T.J. ; Babu, Chitra ; Prabavathy, B.
Abstract :
Hadoop [4] is a popular distributed computing framework, widely used for business data analytics. While Hadoop performs parallel operations on large data sets, it does not co-locate related data by default. However, performance of log processing operations such as indexing, grouping joins and sessionization on Hadoop can be significantly improved if related logs are partitioned and processed as a group. In order to facilitate this, these partitions need to be placed on the same set of nodes in the cluster. To enable this, a grouping key can be used to identify the related logs. CoHadoop [1], an extension of Hadoop, uses this key to co-locate all those files which correspond to the same key. However, it selects the data nodes randomly for every new key. While doing so, the load introduced on every data node of the cluster is not considered. This leads to an imbalance in the cluster. In order to address this issue, a load balanced co-location algorithm is incorporated into CoHadoop and it is termed as CoHadoop++. It achieves load balance in the cluster through optimal selection of data nodes based their load. Experimental results on CoHadoop++ show that the data placement happens in a load balanced manner.
Keywords :
data handling; distributed processing; resource allocation; CoHadoop++; Hadoop distributed file system; business data analytics; data nodes; data placement; distributed computing framework; grouping joins operation; indexing operation; load balanced data co-location; log processing operations; parallel operations; sessionization operation; Bismuth; Clustering algorithms; Distributed databases; Fault diagnosis; Indexes; Co-location; CoHadoop; Load Balancing;
Conference_Titel :
Advanced Computing (ICoAC), 2013 Fifth International Conference on
Conference_Location :
Chennai
Print_ISBN :
978-1-4799-3447-8
DOI :
10.1109/ICoAC.2013.6921934