Asymptotically valid single-stage multiple-comparison procedures

We establish general conditions for the asymptotic validity of single-stage multiple-comparison procedures (MCPs) under the following general framework. There is a finite number of independent alternatives to compare, where each alternative can represent, e.g., a population, treatment, system or stochastic process. Associated with each alternative is an unknown parameter to be estimated, and the goal is to compare the alternatives in terms of the parameters. We establish the MCPs’ asymptotic validity, which occurs as the sample size of each alternative grows large, under two assumptions. First, for each alternative, the estimator of its parameter satisfies a central limit theorem (CLT). Second, we have a consistent estimator of the variance parameter appearing in the CLT. Our framework encompasses comparing means (or other moments) of independent (not necessarily normal) populations, functions of means, quantiles, steady-state means of stochastic processes, and optimal solutions of stochastic approximation by the Kiefer–Wolfowitz algorithm. The MCPs we consider are multiple comparisons with the best, all pairwise comparisons, all contrasts, and all linear combinations, and they allow for unknown and unequal variance parameters and unequal sample sizes across alternatives.