Robust and Efficient Near-Duplicate Documents Detection on P2P Networks

Near-duplicate document detection has received much attention from both industrial and research areas during the recent years. Various methods have been proposed to identify near-duplicate documents that share same content with exception of a small fraction difference. Most of research work are discussed on a centralized system, and focus on the accuracy and efficiency of detection method. However, techniques for detecting near-duplicate documents in a distributed system are very much lacking. In this paper, we propose a Robust and Efficient Near-duplICate document detection on a P2P network (RENDIC), which improves the scalability and performance due to the inheritance of P2P network. Experimental results have demonstrated that our method Rendic achieves a higher effectiveness than that of the existing methods.