A tile-based scalable raster data management system based on HDFS

Author

Zhang, Guangqing ; Xie, Chuanjie ; Shi, Lei ; Du, Yunyan

Author_Institution

State Key Lab. of Resources & Environ. Inf. Syst., Inst. of Geogr. Sci. & Resources Res., Beijing, China

fYear

2012

fDate

15-17 June 2012

Firstpage

1

Lastpage

4

Abstract

Hadoop has become a worldwide popular open source platform for large data analysis in commercial application and Hadoop distributed file system (HDFS) is the core part of it. However, HDFS cannot be used directly for managing raster data, for the geographic location information is involved. In this paper, we describe the implementation of a tile-based scalable raster data management system based on HDFS. While reserving the basic architecture of HDFS, we reorganize the data structure in block, add some additional metadata, design an index data structure in block, keep an overlapping region between adjacent blocks, and offer a compression option for users. Besides, we provide functions for reading the raster data from HDFS in tile stream. These optimizations match the feature of raster data to the architecture of HDFS. MapReduce Applications can be built on the raster data management system.

Keywords

data compression; data structures; distributed databases; geographic information systems; geophysical techniques; geophysics computing; meta data; public domain software; HDFS; Hadoop distributed file system; MapReduce application; file compression option; geographic location information; index data structure; metadata, design; open source platform; optimization; raster data management system; Bandwidth; Computer architecture; Software; Tiles; HDFS; Hadoop; raster data; scalable; tile-based;

fLanguage

English

Publisher

ieee

Conference_Titel

Geoinformatics (GEOINFORMATICS), 2012 20th International Conference on

Conference_Location

Hong Kong

ISSN

2161-024X

Print_ISBN

978-1-4673-1103-8

Type

conf

DOI

10.1109/Geoinformatics.2012.6270280

Filename

6270280