ASPDD: An Efficient and Scalable Framework for Duplication Detection

This paper introduces a framework for duplicate document detection problem that uses an efficient dynamic program called All Pairs Shortest Path in the text collection. Our goal in this work is to investigate the phenomenon and determine the approach that minimizes the impact of duplicates on search results. We show that our approach scales in terms of the number of documents and works well for documents of all domains. We compared our solution to the state of the art and found that our method has produced promising results in addition to improved accuracy of exact duplicate detection, it has also detected partial and neighbor replica. The robustness of the above techniques is demonstrated through a set of experiments using data reflecting real-world degradation effects.