Using quantiles in ranking and selection procedures

A useful performance measure on which to compare manufacturing systems is a quantile of the cycle time distribution. Unfortunately, aside from order statistic estimates, which can require significant data storage, the distribution of quantile estimates has not been shown to be normally distributed, violating a common assumption amongst ranking-and-selection (R&S) procedures. To address this, we provide empirical evidence supporting an approach using the mean of a group of quantile estimates as the comparison measure. The approach is detailed and illustrated through experimentation on four M/M/l queues in which the 0.9 cycle-time quantile is the performance measure. Results in terms of simulation effort and accuracy are reported and compared to results obtained using the macro-replications approach for inducing normality as well as to results obtained by applying R&S procedures to quantile estimates directly. The suggested procedure is shown to provide significant savings in simulation effort while sacrificing very little in accuracy.