A comparison of experimental designs in the development of a neural network simulation metamodel

In this case study, we investigate the effects of experimental design on the development of artificial neural networks as simulation metamodels. A simple, deterministic combat model developed within the paradigm of system dynamics provides the underlying simulation. The neural network metamodels are developed using six different experimental design approaches. These include a traditional full factorial design, a random sampling design, a central composite design, a modified Latin Hypercube design and designs supplemented with domain knowledge. The results from this case study show how much impact the experimental design chosen for the neural network training set can have on the predictive accuracy achieved by the metamodel. We compare the networks in terms of various performance measures. The neural network developed from the modified Latin Hypercube design supplemented with domain knowledge produces the best performance, outperforming networks developed from other designs of the same size.