Performance comparison study of language identification tools for identification of Farsi web pages

With more and more textual information making its way on-line, the importance of web page language identification has become more apparent. Up to now, different methods of language identification for text have been proposed and different tools are prepared according to these methods. But no comparison study on the performance of these tools on web pages has been reported yet. Therefor we decided to evaluate available tools in the field of web page language identification. Our primary desired language is Farsi (Persian), but for acquiring more precision and performing a deeper study we included Arabic, Urdu and English languages, as well. For this purpose, we gathered required information and made two data sets, `Clean Web´ representing clean and noise-free web pages, and `Ordinary Web´ representing normal noisy web pages which we usually visit in our daily web exploring tasks. We evaluated the data sets and tools against desired measures of accuracy and time. Finally we reported the best tools evaluated and discussed the deep effect of noise on their behavior. Results of this study can be applied in vast areas of application such as bootstrapping automatic web page translation, ontology extraction, topic classification, language-specific web crawling...