Statistical analysis and comparison of simulation models of highly dependable systems — An experimental study

The validation of dependability or performance requirements is often done experimentally using simulation experiments. In several applications, the experiments have a binary output which describes whether a requirement is met or not. In highly dependable systems the probability of missing a requirement is 10^-6 or below which implies that statistically significant results have to be computed for binomial distributions with a small probability. In this paper we compare different methods to statistically evaluate simulation experiments with highly dependable systems. Some of the available methods are extended slightly to handle small probabilities and large samples sizes. Different problems like the computation of one or two sided confidence intervals, the comparison of different systems and the ranking of systems are considered.