FENOC: An Ensemble One-Class Learning Framework for Malware Detection

Nowadays, machine learning based methods are among the most popular ones for malware detection. However, most of the previous works use a single type of features, dynamic or static, and take them to build a binary classification model. These methods have limited ability to depict characteristic malware behaviors and suffer from insufficiently sampled benign samples and extremely imbalanced training dataset. In this paper, we present FENOC, an ensemble one-class learning framework for malware detection. FENOC uses hybrid features from multiple semantic layers to ensure comprehensive insights of analyzed programs, and constructs detection model via CosTOC (Cost-sensitive Twin One-class Classifier), a novel one-class learning algorithm, which uses a pair of one-class classifiers to describe malware class and benign program class respectively. CosTOC is more flexible and robust when handling malware detection problems, which is imbalanced and need low false positive rate. Meanwhile, a random subspace ensemble method is used to enhance the generalization ability of CosTOC. Experimental results show that to detect unknown malware, FENOC has a higher detection rate and a lower false positive rate, especially in the situations that training datasets are imbalanced.