Over-fitting behavior of Gaussian unit under Gaussian noise

In the training of neural networks and radial basis function networks under noisy environment, it is important to know how the network over-fits to the noise in the given data since it is directly related to the model selection and regularization problem. In this paper, we firstly derive a probabilistic upper bound for the degree of over-fitting. By applying this result, we consider the over-fitting behavior of a Gaussian unit, which is trained under Gaussian noise, and we show that the probability that the width parameter of the Gaussian unit takes an extremely small value in training under Gaussian noise goes to one as the number of samples goes to infinity.