Top-k future system call prediction based multi-module anomaly detection system

Due to the rapid and continuous development of computer networks, more and more intrusion detection techniques are proposed to protect our systems. However, there is a weak anomaly detection problem among the existing system call based intrusion detection systems: the pattern value range of abnormal system call sequences generated by attacks always overlaps to that by normal behaviors so it is difficult to accurately classify the sequences falling into the overlap area by a unique threshold. Instead of using fuzzy inference, we innovatively solve this problem by proposing a top-k prediction based multi-module (abbreviated as TkPMM) anomaly detection system to enlarge patterns of sequences falling into the overlap area and make them more classifiable. We further develop a scalable linear algorithm called top-k variation of the Viterbi algorithm (called TkVV algorithm) to efficiently predict the top-k most probable future system call sequences. Extensive experimental studies show that TkPMM greatly enhances the intrusion detection accuracy of the existing intrusion detection system by up to 25% in terms of hit rates under small false alarm rate bounds and the complexity of our TkVV algorithm is exponential better than that of the baseline method.