Coping with Variable Latency and Disorder in Distributed Event Streams

Distributed event processing systems often require input streams that are ordered in some predetermined way or that events arrive within a fixed time interval. This order may simply be arrival order or given explicitly on a specific event attribute, such as a timestamp or sequence number. Standing event queries, such as a temporal correlation of events across multiple streams, can block if an input stream incurs latency or may produce an incorrect answer if events fail to arrive within the fixed time interval. In this paper, we describe a stream conditioning mechanism developed for Event Correlation Services (ECS), which is a high performance event processing service for monitoring distributed applications. We show how punctuation messages that carry timestamps can be generated by event sources and then forwarded through the network, to align events from multiple distributed streams and unblock event pattern queries.