Impact of HDFS block size in MapReduce based segmentation and feature extraction algorithm

Apache Hadoop is one of the open source frameworks to process big data. Despite the huge success of Hadoop, it lacks in addressing many of the significant real world problems. And the inability to handle data dependencies efficiently is one of the major issues among those. This paper highlights the data dependency issue in Hadoop framework. This is done using a newly developed MapReduce based segmentation & feature extraction algorithm for very large data dependencies dataset. Data dependency is managed by the variation in HDFS block size. With smaller block size, having larger data dependency and greater parallelism the framework shows uncertain results. As block size is increased, that is by minimizing the data dependency the framework starts to show result stability against the loss of parallelism.