BotCatch: Botnet detection based on coordinated group activities of compromised hosts

Botnets have become one of the major tools used by attackers to perform various malicious activities on the Internet, such as launching distributed denial of service attacks, sending spam, leaking personal information, and so on. In this paper, we present BotCatch, a behavior-based botnet detection system that considers multiple coordinated group activities in the monitored network to identify bot-infected hosts. To achieve this goal, it first identifies suspicious hosts participating in coordinated group activities by an online incremental clustering algorithm and then calculates a negative score for each of the hosts based on several fuzzy membership functions. It then makes an informed decision and identifies a host as bot-infected if its negative score is higher than a threshold. We demonstrate the effectiveness of BotCatch to detect various botnets including HTTP-, IRC-, and P2P-based botnets using a testbed network consisting of some bot-infected hosts. The experimental results show that BotCatch can successfully detect various botnets with a high detection rate while keeping false alarm rate significantly low.