Stream correlation monitoring for uncertainty-aware data processing systems

In several industrial applications, monitoring large-scale infrastructures in order to provide notifications for abnormal behavior is of high significance. For this purpose, the deployment of large-scale sensor networks is the current trend. However, this results in handling vast amounts of low-level, and often unreliable, data, while an efficient and real-time data manipulation is a strong demand. In this paper, we propose an uncertainty-aware data management system capable of monitoring pairwise correlations of large sensor data streams in real-time. An efficient similarity function based on the truncated DFT is employed instead of the typical correlation coefficient to monitor dynamic phenomena for timely alerting notifications, and to guarantee the validity of detected extreme events. Experimental evaluation with a set of real data recorded by distinct sensors in an industrial water desalination plant reveals a high performance of our proposed approach in terms of achieving significantly reduced execution times, along with increased accuracy in detecting highly correlated pairs of sensor data streams.