Debriefing in gaming simulation for research: Opening the black box of the non-trivial machine to assess validity and reliability

Gaming simulation allows for experiments with sociotechnical systems and has as such been employed in the railway sector to study the effects of innovations on robustness and punctuality. Systems work as non-trivial machines and the effect of an innovation on a dependent variable is potentially context, time and history dependent. However, several constraints inhibit the use of validity increasing measures such as repeated runs and increasing sample size. Based on a debriefing framework, insights from qualitative process research and six games with Dutch and UK railway traffic operators, we provide a guide on how to assess and increase reliability and validity. The key is for game players, observers and facilitators to open up the black box and thereby assessing how the innovation brought about any changes, if these changes are insensitive to changes in parameters and if the conclusions hold outside the game.