A new method of data preprocessing and anomaly detection

Data preprocessing including feature extraction is the first significant step in anomaly detection where normal profiles needed to be constructed. This paper defined a sort of traffic flow to be the anomaly event unit of preprocessing, making the data preprocessing module more efficient and robust. Based on TCP flows, the paper introduces a novel methodology to analysis the feature attributes of network traffic flow with some new techniques, including a novel quantization model of TCP states. Integrating with data preprocessing, we construct an anomaly detection algorithm with SOFM and applied the detection frame to DARPA intrusion detection evaluation data. We train SOFM to exploit the normal profile distributions of network traffic, and then the test data with attack-instances embedded is utilized. It is shown that the network attacks are detected with more efficiency and relatively low false alarms.