EXPLORING THE USEFULNESS OF A NONRANDOM HOLDOUT SAMPLE FOR MODEL VALIDATION: WELFARE EFFECTS ON FEMALE BEHAVIOR∗

A particularly challenging use of decision-theoretic models in economics is to forecast the impact of large changes in the environment. The problem we explore in this article is how to gain confidence in a model’s ability to predict the impact of such large changes.We show that an approach to validation and model selection that includes the choice of a “nonrandom holdout sample,” a sample that differs significantly from the estimation sample along the policy dimension that the model is meant to forecast, can be fruitful.