A Hybrid Detection Approach for Zero-Day Polymorphic Shellcodes

Zero-day shellcodes has become a major threat to the Internet with complex obfuscation techniques. However, even the state-of-the-art NIDS has small chances of detecting them because they rely on known signatures. This paper presents hybrid detection for zero-day polymorphic shellcodes (HDPS) against shellcodes using various obfuscations. Our approach employs a heuristic approach to detect return address and filter mass innocent network flows, and then constructs a Markov model to detect the existence and location of executable codes in suspicious flows. Finally, it applies an elaborate approach to detect NOP Sleds in the executable codes. Initial experiments show HDPS detects nearly all types of shellcodes, and the false positive rate approximates zero with low overhead.