On the calibration of Bayesian model choice criteria

Model checking is a crucial part of any statistical analysis. Hierarchical models present special problems because assumptions made about the distribution of unobservable parameters are difficult to check. In this article, we review some approaches to model checking and apply posterior predictive model checking to a hierarchical normal–normal model analysis of data from educational testing experiments in eight schools. Then we carry out a simulation study to investigate the difficulties in model checking for hierarchical models. It turns out that it is very difficult to detect violations of the assumptions made about the population distribution of the parameters unless the extent of violation is huge or the observed data have small standard errors.