Domain Independent Event Analysis for Log Data Reduction

Analyzing the run time behavior of large software systems is a difficult and challenging task. Log analysis has been proposed as a possible solution. However, such an analysis poses unique challenges, mostly due to the volume and diversity of the logged data that is collected, thus making this analysis often intractable for practical purposes. In this paper, we present a log analysis technique that aims to compute a smaller, compared to the original, collection of events that relate to a given analysis objective. The technique is based on computing a similarity score between the logged events and a collection of significant events that we refer to as beacons. The major novelties of the proposed technique are that it is domain independent and that it does not require the use of a pre-existing training data set. The technique has been evaluated against the DARPA Intrusion Detection Evaluation 1999 and the KDD 1999 data sets with promising results.