A Memory-Based Abstraction Approach to Handle Obfuscation in Polymorphic Virus

This paper describes a PhD proposal aiming at dealing with obfuscation in polymorphic virus. The major characteristic of polymorphic virus is the capability of infinitely self-modifying when infecting victim programs. It makes the traditional signature-based virus detection technique ineffective since this approach needs to collect all of signature instances. A recent emerging approach to counter this problem is abstracting the program from binary level, then extracting an abstracted model for further analysis. The most common model to be extracted is perhaps the control flow graph (CFG) of the binary program. However, this control-based abstraction approach is currently suffering from some advanced obfuscation techniques which change not only the signatures but also modify significantly the control flow of the programs. Thus, the control analysis will become quickly too complicated. Hence, we propose a novel approach of abstracting the binary code based on memory states. This approach allows us to detect useless instructions which are part of obfuscation code. Moreover, for the next step, our approach can be extended as new efficient technique for virus detection based on common abstracted pattern.