A self-evolving anomaly detection framework for developing highly dependable utility clouds

Utility clouds continue to grow in scale and in the complexity of their components and interactions, which introduces a key challenge to failure and resource management for highly dependable cloud computing. Autonomic anomaly detection is a crucial technique for understanding emergent, cloud-wide phenomena and self-managing cloud resources for system-level dependability assurance. To identify anomalies, we need to monitor the system execution and collect health-related runtime performance data. These data are usually unlabeled and a prior failure history is not always available in production systems, especially for newly deployed or managed utility clouds. In this paper, we present a self-evolving anomaly detection framework with mechanisms for dependability assurance in utility clouds. No prior failure history is required. The detector self-evolves by recursively exploring newly generated verified detection results for future anomaly identification. Statistical learning technologies are exploited in detector determination and working dataset selection. Experimental results in an institute-wide cloud computing system show that the detection accuracy improves as it evolves. With self-evolvement, the detector can achieve 92.1% detection sensitivity and 83.8% detection specificity, which makes it well suitable for building highly dependable utility clouds.