Intelligent Similarity Joins for Big Data Integration

With the increasing amount of data, the record linkage has become a challenge for big data integration. Similarity join is an efficient approach to address the record linkage, but it is hardly achieved by the single node environment. In this paper, we propose a framework based on MapReduce for set similarity join. The techniques of framework improve the efficiency from two aspects: reducing candidate pairs and load balance. In reducing candidate pairs, we propose algorithms that combines multiple filtering principles to reduce the amount of candidate pairs. It includes length filter, prefix filter and position filter. The techniques for load balance are used to address the skew data and decrease the replication transfer volume. Experimental results on real dataset show that our approaches can achieve the speed-up over previous algorithms on big data.