The realization of the distributed search engine on cloud platform

With the rapid development of the Internet and websites, the amount of web data increases exponentially, which leads to great challenges of the traditional centralized search engine in the real-time search, response speed and the storage of mass pages. Cloud computer, which can integrate resources of many PCs to provide distributed storage and parallel computing, has many advantages in dealing with mass data. As a result, a distributed search engine on cloud platform is proposed in this paper. It consists of three main modules: crawling, indexing and retrieving. Furthermore, it also presents visual user interaction interfaces. The search engine is implemented on Hadoop, with Hadoop Distributed File System (HDFS) for storage and parallel programming model MapReduce to realize indexing and retrieving functions. The search engine on Hadoop can store mass pages with many cheap machines, retrieve the wanted information from mass data as shown in function test and reduce query time as shown in performance test, so it is economical, accurate and efficient.