Title :
Racewalk: Fast Instruction Frequency Analysis and Classification for Shellcode Detection in Network Flow
Author :
Gamayunov, Dennis ; Quan, Nguyen Thoi Minh ; Sakharov, Fedor ; Toroshchin, Edward
Author_Institution :
Dept. of Comput. Math. & Cybern., Moscow State Univ., Moscow, Russia
Abstract :
Memory corruption attacks still play a significant role in present cybercrime activities, being one of the keystones for worm, virus propagation and building botnets. Moreover,recent disclosures of widespread networking equipment vulnerabilities show that the problem is unlikely to fade away in the near future. The subject of this paper is NOP-sled detection - one of the approaches for detecting malicious code in network flow. NOP-sled is a quite common shellcode preamble used in memory corruption attacks to increase the probability of successful target exploitation. We propose a significant modification of the Stride algorithm which has linear computational complexity and runs over 10 times faster than original Stride and a novel approach for NOP-sled detection using IA-32 instruction frequency analysis and SVM-based classification, which gives significantly less false positives then existing algorithms. Evaluation with Metasploit Framework, CLET,ecl-poly and ADMmutate shows that various NOP-sleds provided by existing shellcode generators have instructions frequency peculiarities, which allow to distinguish between sleds and normal network data with high accuracy while reducing the false positives rate and operating close to 1Gbps speed.
Keywords :
computer network security; pattern classification; support vector machines; IA-32 instruction frequency analysis; NOP-sled detection; Racewalk; SVM-based classification; Stride algorithm; botnets; instruction frequency analysis; instruction frequency classification; linear computational complexity; memory corruption attacks; network flow; shellcode detection; support vector machines; virus propagation; worm; Computer aided instruction; Computer crime; Computer networks; Computer worms; Frequency; Internet; Intrusion detection; Operating systems; Permission; Support vector machines; SVM; instruction frequency analysis; intrusion detection; intrusion prevention; metamorphism; polymorphism; shellcode; support vector machine;
Conference_Titel :
Computer Network Defense (EC2ND), 2009 European Conference on
Conference_Location :
Milan
Print_ISBN :
978-1-4244-6049-6
DOI :
10.1109/EC2ND.2009.9