Title :
A tile-based scalable raster data management system based on HDFS
Author :
Zhang, Guangqing ; Xie, Chuanjie ; Shi, Lei ; Du, Yunyan
Author_Institution :
State Key Lab. of Resources & Environ. Inf. Syst., Inst. of Geogr. Sci. & Resources Res., Beijing, China
Abstract :
Hadoop has become a worldwide popular open source platform for large data analysis in commercial application and Hadoop distributed file system (HDFS) is the core part of it. However, HDFS cannot be used directly for managing raster data, for the geographic location information is involved. In this paper, we describe the implementation of a tile-based scalable raster data management system based on HDFS. While reserving the basic architecture of HDFS, we reorganize the data structure in block, add some additional metadata, design an index data structure in block, keep an overlapping region between adjacent blocks, and offer a compression option for users. Besides, we provide functions for reading the raster data from HDFS in tile stream. These optimizations match the feature of raster data to the architecture of HDFS. MapReduce Applications can be built on the raster data management system.
Keywords :
data compression; data structures; distributed databases; geographic information systems; geophysical techniques; geophysics computing; meta data; public domain software; HDFS; Hadoop distributed file system; MapReduce application; file compression option; geographic location information; index data structure; metadata, design; open source platform; optimization; raster data management system; Bandwidth; Computer architecture; Software; Tiles; HDFS; Hadoop; raster data; scalable; tile-based;
Conference_Titel :
Geoinformatics (GEOINFORMATICS), 2012 20th International Conference on
Conference_Location :
Hong Kong
Print_ISBN :
978-1-4673-1103-8
DOI :
10.1109/Geoinformatics.2012.6270280
