A Distributed File System for Frequency Reading of Various File Sizes

Author

Pengfei Ma ; Yanshen Yin ; Chao Lan ; Yong Zhang ; Chunxiao Xing

Author_Institution

Dept. of Comput. Sci. & Technol., Tsinghua Univ., Beijing, China

fYear

2013

Firstpage

339

Lastpage

344

Abstract

The Web File Management System (WFMS) is a distributed file system, which is designed for frequency reading of various file sizes. It consists of one single master and multi data servers. We use blocks to store data, which contains different size of chunks. The block/chunk mechanism achieves nearly 100% utilization of block. We also design hot block to raise the reading speed of frequency blocks through increasing the replicas automatically. WFMS works on the web exposing Restful services and providing authority control. Besides, we implement our search engine with Lucene which provides query service by file name, file size, file type, date, owner and so on. Until now, our system has stored over 40TB data which contains more than 2 million files and it has worked stably for two years.

Keywords

Internet; authorisation; query processing; replicated databases; search engines; storage management; Lucene; Restful services; WFMS; Web file management system; authority control; block/chunk mechanism; data storage; distributed file system; file sizes; frequency blocks; frequency reading; multidata server; query service; reading speed; search engine; single master server; Access control; Distributed databases; File systems; Indexes; Reliability; Schedules; Servers; WFMS; distributed file system; file system;

fLanguage

English

Publisher

ieee

Conference_Titel

Web Information System and Application Conference (WISA), 2013 10th

Conference_Location

Yangzhou

Print_ISBN

978-1-4799-3218-4

Type

conf

DOI

10.1109/WISA.2013.70

Filename

6778661