Title :
The realization of green storage in Hadoop
Author :
Qiao Zhu ; Li Miao
Author_Institution :
Dept. of Comput. Sci. & Eng., Hunan Univ., Changsha, China
Abstract :
Hadoop has been successful at harnessing expansive data-centers resources for large-scale data analysis. However, their effect on data-centers energy efficiency has not scrutinized completely. The energy consumption of Hadoop Distributed File System in data-centers accounts for a great part of total cost of ownership and disk is the main storage media for clusters. Analysis of the interactions between clusters with disks when running a Hadoop application showed a disk idle time when shuffle to memory, which can be used to guide a simple green storage algorithm for Hadoop cluster. The algorithm simulation results with Terasort for 10G data in a cluster with 11 nodes can save 2.47WH.
Keywords :
computer centres; data analysis; distributed databases; parallel processing; storage management; Hadoop; Terasort; data centers resources; distributed file system; green storage; large-scale data analysis; Blogs; Buffer storage; Data models; Green products; Reliability; Disk; Energy; Green storage; Hadoop;
Conference_Titel :
Cloud Computing and Internet of Things (CCIOT), 2014 International Conference on
Print_ISBN :
978-1-4799-4765-2
DOI :
10.1109/CCIOT.2014.7062512
