Partitioner: A Distributed HDFS Metadata Server Cluster

Author

Ruini Xue ; Lixiang Ao ; Shengli Gao ; Zhongyang Guan ; Lupeng Lian

Author_Institution

Sch. of Comput. Sci. & Eng., Univ. of Electron. Sci. & Technol. of China, Chengdu, China

fYear

2014

fDate

13-15 Oct. 2014

Firstpage

167

Lastpage

174

Abstract

The Hadoop Distributed File System (HDFS) provides scalable, high throughput and fault-tolerant file service for Hadoop distributed frameworks. However, its single Name Node architecture can lead the metadata server to be a performance bottleneck and a single point of failure (SPOF). In this study, we designed and implemented Partitioner, a distributed HDFS metadata server cluster, which can unite multiple Name Nodes and provide single unique namespace for clients. To support the distributed metadata management, a novel sub tree based metadata partitioning and lookup scheme is proposed. Experiment results indicated that this method can eliminate SPOF while expanding metadata capacity, resulting in improved scalability and availability of the distributed file system.

Keywords

data handling; distributed databases; fault tolerance; meta data; parallel processing; Hadoop distributed file system; SPOF; distributed HDFS meta data server cluster; distributed meta data management; fault-tolerant file service; single name node architecture; single point of failure; Availability; Computer architecture; Distributed databases; Load management; Monitoring; Scalability; Servers; HDFS; distributed file system; metadata management; scalability;

fLanguage

English

Publisher

ieee

Conference_Titel

Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC), 2014 International Conference on

Conference_Location

Shanghai

Print_ISBN

978-1-4799-6235-8

Type

conf

DOI

10.1109/CyberC.2014.38

Filename

6984301