An Empirical Exploration of Black-Box Performance Models for Storage Systems

The effectiveness of automatic storage management depends on the accuracy of the storage performance models that are used for making resource allocation decisions. Several approaches have been proposed for modeling. Black-box approaches are the most promising in real-world storage systems because they require minimal device specific information, and are self-evolving with respect to changes in the system. However, blackbox techniques have been traditionally considered inaccurate and non-converging in real-world systems. This paper evaluates a popular off-the-shelf black-box technique for modeling a real-world storage environment. We measured the accuracy of performance predictions in single workload and multiple workload environments. We also analyzed accuracy of different performance metrics namely throughput, latency, and detection of saturation state. By empirically exploring improvements for the model accuracy, we discovered that by limiting the component model training for the nonsaturated zone only and by taking into account the number of outstanding IO requests, the error rate of the throughput model is 4.5% and the latency model is 19.3%. We also discovered that for systems with multiple workloads, it is necessary to consider access characteristics of each workload as input parameters for the model. Lastly, we report results on the sensitivity of model accuracy as a function of the amount of bootstrapping data.