Understanding Vicious Cycles in Server Clusters

In this paper, we present an automated on-line service for troubleshooting performance problems in server clusters caused by unintended vicious cycles. The tool complements a large volume of prior performance troubleshooting and diagnostic literature for server farms that identifies problems arising due to resource bottlenecks or failed components. We show that unintended interactions between components in large-scale systems can cause performance problems even in the absence of bottlenecks or failures. Our tool leverages discriminative sequence mining to identify anomalous sequences of events that are candidates for blame for the performance problem. The tool looks for patterns consistent with "vicious cycles" or unstable behavior, as such patterns, when present, are most likely to be problematic. It highlights candidates that are semantically conflicting, such as those arising when different performance management mechanisms make adjustments in conflicting directions. Our approach offers two key advantages in performance troubleshooting. First, it does not require detailed prior knowledge of the underlying system to diagnose the problem. Second, contrary to simple statistical techniques, such as correlation analysis, that work well for continuous variables, our scheme can also identify chains of events (labels) that may explain the root cause of a problem. Our service is deployed on a web server testbed of 17 machines. To make the comparison of our scheme to prior work more concrete, we first reproduce two real-life problem scenarios reported in earlier literature, then explore a third, new case study. In all cases, our tool reports the patterns that explain the cause of the problem without requiring detailed a priori knowledge.