Adaptive real-time anomaly detection with improved index and ability to forget

Anomaly detection in IP networks, detection of deviations from what is considered normal, is an important complement to misuse detection based on known attack descriptions. Performing anomaly detection in real-time places hard requirements on the algorithms used. First, to deal with the massive data volumes one needs to have efficient data structures and indexing mechanisms. Secondly, the dynamic nature of today´s information networks makes the characterization of normal requests and services difficult. What is considered as normal during some time interval may be classified as abnormal in a new context, and vice versa. These factors make many proposed data mining techniques less suitable for real-time intrusion detection. In this paper we extend ADW ICE, anomaly detection with fast incremental clustering. Accuracy of ADW ICE classifications is improved by introducing a new grid-based index, and its ability to build models incrementally is extended by introducing forgetting. We evaluate the technique on the KDD data set as well as on data from a real (telecom) IP test network. The experiments show good detection quality and illustrate the usefulness of adapting to normality.