A Method for Detecting Unknown Malicious Executables

We present a method for detecting new malicious executables, which comprise the following steps: (a) in an offline training phase, finding a set of (not necessary consecutive) system call sequences that are characteristic only to malicious files, when such malicious files are executed, and storing said sequences in a database; (b) in a real time detection phase, for each running executable, continuously monitoring its issued system calls and comparing with the stored sequences of system calls within the database to determine whether there exists a match between a portion of the sequence of the run-time system calls and one or more of the database sequences, and when such a match is found, declaring said executable as malicious. We have evaluated our method and the preliminary results are promising and justify the use of system calls sequences for the purpose of detection of new malicious executables.