Comparison of Bayesian and frequentist assessments of uncertainty for selecting the best system

An important problem in discrete event stochastic simulation is the selection of the best system from a finite set of alternatives. There are many techniques for ranking and selection and multiple comparisons discussed in the literature. Most procedures employ classical frequentist approaches, although there has been recent attention to Bayesian methods. We compare Bayesian and frequentist assessments of unknown means of simulation output. First, we present a Bayesian formulation for describing the probability that a system is the best, given prior information and simulation output. This formulation provides a measure of evidence that a given system is best when there are two or more systems, with either independent or common random numbers, with known or unknown variance and covariance for the simulation output, given a Gaussian output assumption. Many, but not all frequentist assessments are shown to be derivable from assumptions of normality of simulation output when certain limits are taken. So we compare Bayesian probability of correct selection (P(CS)) with frequentist P-value as a measure of evidence that the best system is selected under normality assumptions