Covariance-based weighting for optimal combination of model predictions

This paper introduces a method for calculating the covariance between different neural network solutions. It is based on a generalisation of the delta method for calculating the network Hessian and generates what we call the `cross-covariance´ matrix (its inverse is the `cross-Hessian´). Using this matrix we are able to estimate the covariance between network predictions at each point in input space, using training data alone. Whilst this is a significant result in itself we have also applied the method to the problem of finding optimal linear combinations of models. This results in a `covariance-based´ weighted committee, where the weights are input-dependent. If the individual networks are unbiased then the covariance-based weighted committee is optimal in the sense of minimum expected prediction error