A Large Scale URL Verification Pipeline Using Hadoop

Data quality is a key element for local search and advertising. Inaccurate, out-of-date or missing information causes an unpleasant search experience for users and affects competitiveness of service providers. This paper addresses the problem of evaluating link quality for business listings in local search and online advertising domain. We introduce a novel system where we apply data mining technologies on a Hadoop-based platform to provide an efficient and highly scalable solution for the problem. Due to various reasons, links associated with business listings do not always point to their business websites. Possible noises include parked domain, broken links, third-party advertisers, irrelevant websites etc. To detect above noises and improve link quality, we formulate this problem as a binary classification problem: whether a given URL is the business website of the associated listing. Experiments conducted on real-world data show that our system can verify millions of business listings against about 100 million web pages in a couple of hours with 93% classification accuracy.