Analysis of Log Files Intersections for Security Enhancement

In this paper, we discuss our research in developing general and systematic methods for intrusion prevention. The key idea is to use data mining techniques to discover regular patterns of system features that describe program and user behavior. Server systems invariably write detailed activity logs whose value is useful in detecting intrusion. Unfortunately, production volumes overwhelm the capacity and manageability of traditional approach. This paper discusses the issues involving large-scale log processing that helps analyze log records. In this paper we propose to analyze intersections of log files that come from different applications and firewalls installed on one computer, and intersections resulting from log files coming from different computers. Intersections of log files are substantially smaller than full logs and consist of records that indicate abnormalities in accessing single computer or set of computers. The paper concludes with some lessons we learned in building the system