A Novel Anomaly Detection Approach for Executable Program Security

Anomaly detection of executable program is a security detection solution that examines whether security violation issues exist in programs. The paper presents a novel anomaly detection approach for executable program security (ADEPS), which monitors program executions and detects anomalous program behaviors. Through reverse analysis of executable program, critical behavior monitoring points can be extracted from binary code sequences and memory space. A hybrid neural network model is proposed to detect abnormal attacks and classify detected attacks from actual program behaviors. The experimental results demonstrate that the proposed approach can effectively and accurately perform anomaly detection.