ISMCS: An intelligent instruction sequence based malware categorization system

Recently, automated malware (e.g., viruses, backdoors, spyware, Trojans and worms) categorization methods and an industry-wide naming convention have been the computer security topics that are of great interest. Resting on the analysis of function based instruction sequence, we develop an intelligent instruction sequence based malware categorization system (ISMCS) using a novel weighted subspace clustering method. ISMCS is an integrated system consisting of three major modules: feature exactor, malware categorizer using weighted subspace clustering method and malware signature generator. ISMCS can not only effectively categorize malwares to different families, but also automatically generate the unify signature for every family. Promising experimental results demonstrate that the effectiveness of our ISMCS system outperform other existing malware categorization methods, such as K-Means and hierarchical clustering algorithms.