Static Detection of API-Calling Behavior from Malicious Binary Executables

The broad spread of malware in recent years has presented a serious threat to our world. Because Windows API-calling sequence usually reflects the vicious behavior in a piece of particular code, more and more AV researchers like to detect malware based on API-calling behavior analysis. However, a great many of techniques, such as obfuscation, have been used by malware writers to evade this type of detection. These techniques makes the discovery of API-calling behavior become more complex than before. In this paper, we illustrate some methods which are commonly used by malware writers to obscure their API-calling behavior when they write their malware in assembly language. After that, we propose a new approach, which is more universal for capturing API-calling behaviors in Windows platform. This approach involves three databases and some special instruction patterns. Experimental results show that using this approach to extract API-calling behaviors from malicious executables and their variants is favorable and effective.