Big Data Processing with Probabilistic Latent Semantic Analysis on MapReduce

Probabilistic Latent Semantic Analysis (PLSA) is a powerful statistical technique to analyze co-occurrence data, it has wide usage in information processing, ranging from information retrieval, information filtering, text processing automation, to natural language processing, and related areas. However, it has very high time and space complexity to train PLSA model on big data. Researchers have been trying to solve this problem using parallel means. But their approaches only partially reduce the time complexity, the main memory in the compute process still needs to load a large amount of data. In order to solve the scalability problem of data, a parallel method to train PLSA is proposed by adapting the traditional EM algorithm into MapReduce a computing framework for processing vast amounts of data in-parallel on clusters. In this way, the main memory in each computer just needs to load part of the dataset. This method can reduce time and space complexity simultaneously. Results show that this method can deal with large datasets efficiently.