Web bots detection using Particle Swarm Optimization based clustering

Optimization based techniques have emerged as important methods to tackle the problems of efficiency and accuracy in data mining. One of the current application areas is outlier detection that has not been fully explored yet but has enormous potential. Web bots are an example of outliers, which can be found in the web usage analysis process. Web bot requests are different from a genuine web user as web bots crawl large numbers of pages in a very short time. If web bots remains undetected they can skew the analysis process which can result in incorrect patterns that can cause wrong decisions. In this paper we use one of the popular Swarm Intelligence (SI) based techniques called Particle Swarm Optimization (PSO) to detect web bots among genuine user requests. We use our Particle Swarm Optimization (PSO) based clustering algorithm, Hierarchical Particle Swarm Optimization based clustering (HPSO-clustering) to cluster the web-usage data and detect the abnormal behaviour caused by the web bots. We present the results of detection which shows that our proposed approach is capable of detecting such abnormal behaviour. We then compare the characteristics of the detected web bots with genuine web-users using cross validation.