High-Performance Distributed Indexing and Retrieval for Large Volume Traffic Log Datasets on the Cloud

In this paper, we present a high-performance distributed system for storage, indexing and retrieval for large volume web traffic log datasets. This system is Based on the open source Map Reduce framework Hadoop and extends the functionality of Hadoop. We mainly focus on three noteworthy aspects of our work: the approach of large datasets storage on the Hadoop Distributed File System (HDFS), the appropriate indexing algorithm for large distributed datasets, a distributed retrieval architecture built on Hadoop. It has been proved that our system is efficient and the query response latency approach real time compared with HBase, a distributed, sparse, NoSQL database.