Malicious Shellcode Detection with Virtual Memory Snapshots

Malicious shellcodes are segments of binary code disguised as normal input data. Such shellcodes can be injected into a target process´s virtual memory. They overwrite the process´s return addresses and hijack control flow. Detecting and filtering out such shellcodes is vital to prevent damage. In this paper, we propose a new malicious shellcode detection methodology in which we take snapshots of the process´s virtual memory before input data are consumed, and feed the snapshots to a malicious shellcode detector. These snapshots are used to instantiate a runtime environment that emulates the target process´s input data consumption to monitor shellcodes´ behaviors. The snapshots can also be used to examine the system calls that shellcodes invoke, these system call parameters, and the process´s execution flow. We implement a prototype system in Debian Linux with kernel version 2.6.26. Our extensive experiments with real traces and thousands of malicious shellcodes illustrate our system´s performance with low overhead and few false negatives and few false positives.