Outlier Detection in Large-Scale Sensor Network Data Using Shrinkage Estimators

Outlier detection is a frequently encountered technology challenge for many diverse applications in sensor networks, and remains an open problem in general. There are two major difficulties of developing outlier detectors with sensor data. One is the inevitable multi-source identification, the other is the effective inference when discovering information from unknown structured large-scale data. It is even more interesting and challenging with limited observation, since conventional data analysis requires many samples to achieve a satisfactory performance. In this paper, we systematically develop effective and efficient outlier identifiers in parametric and non-parametric ways using shrinkage methodology, like the James-Stein estimator, as the post-processor. We show the superiority of our approach, particularly for the large-scale situations. We further supply a water- filling type algorithm to obtain the asymptotic optimal method for a general class of shrinkage estimators, for wide applications of data analysis.