On the use of weighted syslog time series for anomaly detection

Finding unusual (anomalous) events in a network is a crucial task for network operation. The collection and analysis of the log messages from the network devices (i.e., router and switch) are a common method. However, the detection of the anomalous events is painful because of a huge amount and many types of log messages; the important event messages are sometimes hidden in a large number of relatively less important and usual event messages. Many researches have been devoted to find out anomalies accurately and quickly by using time series of log message. To construct the log time series for anomaly detection in order to highlight such hidden anomalies, this paper focuses on the effectiveness of using a global weight that is based on a global appearance of a message type in the all data set. We introduce two types of the global weight called residual inverse document frequency (RIDF) and entropy that are well-known method in information retrieval field. Then, we evaluate the performance improvement due to the global weights with a wavelet-based abrupt change detection algorithm through 1-year long collection of core router log messages taken from a Japanese R&E network. Our main findings are (1) the global weights assign a low weight to less important and frequently appeared event messages, (2) they highlight unusual events by removing temporal correlation (i.e., periodic trend), and (3) the wavelet-based detection algorithm more accurately alarms the anomalous time period with the weighted time series.