Event Indexing and Searching for High Volumes of Event Streams in the Cloud

Deployed software applications use log files to keep a record of system events. Log analysis provides support for system administrators to gain the knowledge of system health and behavior. As a result, the ability to efficiently search for patterns in historical events has become a major requirement for timely analysis. Enterprise systems today produce high volumes of log data, regularly in the order of thousands of events per second, which requires to build inverted indexes for quick data retrieval. However, current inverted indexing techniques are rarely designed to handle high volumes of dynamic stream data and often resource consuming. We propose an efficient indexing solution, which reduces the necessary resources by employing bloom filter techniques. The solution builds a generic indexing engine for the Run Time Correlation Engine logging framework to achieve efficient monitoring in the Cloud. In particular, our solution is able to deliver significant performance improvement over existing indexing engines.