Analysis of simulation factorial experiments by EDF resample statistics

The output from simulation factorial experiments can be complex and may not be amenable to standard methods of estimation like ANOVA. We consider the situation where the simulation output may not satisfy normality assumptions, but more importantly, where there may be differences in output at different factor combinations, but these are not simply differences in means. We show that EDF statistics can provide a similar but potentially more sensitive analysis to that provided by ANOVA. Moreover, we show that with the use of resampling, we can generate accurate critical values for tests of hypothesis under much weaker conditions than those required for ANOVA tests. The method is illustrated with an example based on an actual simulation experiment comparing two methods of operating a production facility under different production levels