Language based web crawling on big data

Online textual and visual data that are created and used by web users have been increasing dramatically and continually. This increase has caused the need for easy and fast access to online data and facilitated the development of alternative means of access to this data. Nowadays, web crawlers are the most efficient and popular tools used for accessing big volumes of data available on the web. In this paper, a web crawler which works on a distributed Hadoop cluster for crawling web pages with content of a predefined language is described. A language identification tool is developed for enabling the system to focus only on a specific language. In this study, the accuracy of the language identification tool is evaluated on a small data set (consisting of 4729 web pages). The performance of the focused web crawling system is reported on a big data set of 86 million web pages containing Turkish content.