Clustering of Similar Malware Behavior via Structural Host-Sequence Comparison

Malware (malicious software) is used by attackers to gain access to end-users´ computing devices with the aim of performing malicious actions, such as sending spam, downloading malicious files, and stealing private information. Furthermore, malicious actions result in the infection of many machines at considerable monetary expense to affected end-users. Although anti-malware companies can detect malware, they generally rely on signature-based methods. Because attackers use automated malware generation tools to generate new or modified malware that leads to increased instances and variants of malware, variants of the same malware class can often evade such methods. In this study, we propose a system based on structural sequence comparison and probabilistic similarity measures for detecting variants of a malware class. We believe that the structural sequence of malware behavior can detect variants of the same malware class by profiling the relationship between different behavior patterns. Furthermore, probabilistic similarity measures are used for discovering the behavior patterns of the sequential relation and finding very similar behavior models of malware classes. The proposed system, which incorporates a structural sequence mechanism and probabilistic similarity measurement, can be helpful for detecting and discovering variants of a malware class. The results of our experiment show that the proposed system detects variants of a malware class more effectively than a two-gram sequence does, and it lowers the effect and weight of behavior patterns, with higher precision and F-measure in variants of malware detection than the approach based on a two-gram sequence and normalized embedding function.