Towards a Theory of Generalizing System Call Representation for In-Execution Malware Detection

The major contribution of this paper is two-folds: (1) we present our novel variable-length system call representation scheme compared to existing fixed- length sequence schemes, and (2) using this representation, we present our in-execution malware detector that can not only identify zero-day malware without any a priori knowledge but can also detect a malicious process while it is executing. Our representation scheme - a more generalized version of n-gram - can be visualized in a k-dimensional hyperspace in which processes move depending upon their sequence of system calls. The process marks its impact in space by generating hyper-grams that are later used to evaluate an unknown process according to their profile. The proposed technique is evaluated on a real world dataset extracted from a Linux System. The results of our analysis show that our in-execution malware detector with hyper- gram representation achieves low processing overheads and improved detection accuracies as compared to conventional n-grams.