Automatic functionality detection in behavior-based IDS

Detection of malicious functionalities presents an effective way to detect malware in behavior-based IDS. A technology including the utilization of Colored Petri Nets for the generalized description and consequent detection of specific malicious functionalities from system call data has been previously developed, verified and presented. A successful effort was made to neutralize possible attempts to obfuscate this approach. Nevertheless, the approach has two major drawbacks. First, target functionalities have to be initially specified by an expert, which is a time consuming, sometimes subjective and error prone process. Second, the identification of typical functionalities indicative of malicious programs is not generally straightforward and requires reverse engineering and careful study of many instances of malware. Our paper addresses these drawbacks, clearing the way for a full-scale practical application of this technology. We utilized graph mining and graph similarity assessment algorithms for processing system call data resulting in automatic extraction of functionalities from system call data. This enabled us to identify sets of functionalities suggesting software maliciousness and construct a general obfuscation-resilient malware detector. The paper presents the results of the implementation and testing of the described technologies on the computer network testbed.