An optimized cloud based big data processing mechanism using Self-Organizing Map in Hadoop environments

Large scale searching problems such as searching for a particular tag in a set of web pages are always challenging. Distributed implementation of such searching algorithms are used in different distributed systems that includes Grid, Hadoop etc. In general, hadoop framework uses the Hadoop Distributed File System (HDFS) for all kinds of data processing. But the efficiency regarding the time in a distributed environment for data processing without any optimized algorithm is comparatively low. In this work, these problems are addressed. Searching using MapReduce paradigm is considered for implementing proposed scheme in the popular Open Source Cloud computing platform Hadoop and a neural network optimization algorithm named Self Organizing Maps (SOM). The processing speed got increased as the number of nodes in the Hadoop environment increases.