HDFS+: Concurrent Writes Improvements for HDFS

HDFS is a popular distributed file system which provides high scalability and throughput. It lacks built-in support for multi-source data generating, which arise naturally in many applications including log mining, data analysis etc. There needs a data collection step before analysis in basic HDFS environment because of many data are in local disk, such as log. We proposed a solution which can compose many existent files to a single file and it is suitable for concurrent writes by many data producers. Programs only have to implements data processing against one single file without a data collection step when data analysis. We implemented HDFS+ by modifying existent HDFS, and evaluated with applications including log analysis. Our results show great throughput improvements in data concurrent writes. HDFS+ vastly simplifies the data collecting steps in data analysis procedure.