A Novel Hybrid Method for Polymorphic Worm Detection

Since worms have become a major threat of cybersecurity, several detection approaches have been proposed to detect them. However, attackers have exploited state-of-the-art techniques to evade these detection systems, such as polymorphism and metamorphism, making existing systems ineffective. In this paper, we propose a hybrid method for the detection of polymorphic worms. It uses improved reverse sequential hypothesis testing (RSHT) to detect portscans which are routinely used to find vulnerable hosts to compromise. Then a CPU emulator is used to execute every possible instruction sequence in suspicious traffic and determine whether it is an exploit code. We implemented a prototype and tested it using real polymorphic worms. Initial experimental results show that our approach is effective with high accuracy.