Title :
Probabilistic reasoning for streaming anomaly detection
Author :
Carter, Kevin M. ; Streilein, William W.
Author_Institution :
MIT Lincoln Lab., Lexington, MA, USA
Abstract :
In many applications it is necessary to determine whether an observation from an incoming high-volume data stream matches expectations or is anomalous. A common method for performing this task is to use an Exponentially Weighted Moving Average (EWMA), which smooths out the minor variations of the data stream. While EWMA is efficient at processing high-rate streams, it can be very volatile to abrupt transient changes in the data, losing utility for appropriately detecting anomalies. In this paper we present a probabilistic approach to EWMA which dynamically adapts the weighting based on the observation probability. This results in robustness to data anomalies yet quick adaptability to distributional data shifts.
Keywords :
inference mechanisms; probability; security of data; abrupt transient change; distributional data shift; exponentially weighted moving average; high volume data stream; observation probability; probabilistic approach; probabilistic reasoning; streaming anomaly detection; Data models; Predictive models; Probabilistic logic; Robustness; Standards; Storage area networks; Transient analysis; Anomaly detection; information security; predictive models; statistical learning; time series analysis;
Conference_Titel :
Statistical Signal Processing Workshop (SSP), 2012 IEEE
Conference_Location :
Ann Arbor, MI
Print_ISBN :
978-1-4673-0182-4
Electronic_ISBN :
pending
DOI :
10.1109/SSP.2012.6319708
