Searching for Widespread Events in Large Networked Systems by Cooperative Monitoring

Searching for widespread events in large networks is a fundamental function that underlies many important applications of distributed anomaly detection, traffic measurement, online data mining, etc. This function can be performed by a cooperative monitoring system consisting of a central coordinator and a number of monitors that are deployed at a set of vantage points. We formulate a network primitive function, called multi-monitor joint detection, which is to find the common events observed by all or a given subset of monitors during each measurement period. It is a challenging problem because large-scale cooperative monitoring can generate tremendous communication overhead. Therefore, it is critical to design a solution for multi-monitor joint detection which controls communication overhead to a low level. We thoroughly examine existing techniques that may be applied, and identify their performance limitations. We then propose two new techniques, called combinable filters and progressive filtering, which address the performance limitations from different angles. We formally prove the correctness of our new solutions based on a probabilistic joint detection model. Numerical evaluation shows that our best solution achieves an overhead reduction in the range of 63% to 91% over the Bloom filter solution under various simulation settings when the number of monitors is 10 or more.