A collaborative filtering recommendation engine in a distributed environment

The tremendous increase in information available over the Internet has created a challenge in searching of useful information, therefore intelligent approaches are needed to provide users to efficiently locate and retrieve information from the Web. In recent times recommender systems, recommend everything from movies, books, music, restaurant, news to jokes. Collaborative filtering (CF) algorithms are one of the most successful recommendation techniques which present information on items and products that are according to user´s interest. There are two methods in CF, user-based CF and item-based CF. Former finds a certain user´s interests by finding other users who have similar interests whereas item based CF looks into a set of items rated by all users and computes how similar they are to the target item under recommendation. This paper aims to develop a model by splitting the costly computations in CF algorithms into three Map-Reduce phases. Further, each of these phases can be executed independently on different nodes in parallel. To compute the similarity, the Pearson correlation algorithm is used, which measures the how two items linearly relate to each other, giving a value between -1 and +1 inclusive. In addition, this paper compares the implementation of item based and user based CF algorithm on map-reduce framework. Experimental results showed that the running time of the algorithm improves by approximately 30% with every addition of a node, into a Hadoop cluster. However, item-based CF showed better scalability than user-based CF.