On the problem in model selection of neural network regression in overrealizable scenario

In this article, we analyze the expected training error and the expected generalization error in a special case of overrealizable scenario, in which output data is a Gaussian noise sequence. Firstly, we derived the upper bound of the expected training error of a network, which is independent of input probability distributions. Secondly, based on the first result, we derived the lower bound for the expected generalization error of a network, provided that the inputs are not stochastic. From the first result, it is clear that we should evaluate the degree of overfitting of a network to noise component in data more larger than the evaluation in NIC. From the second result, the expected generalization error, which is directly associated with the model selection criterion, is larger than in NIC. These results suggest that the model selection criterion in overrealizable scenario will be larger than NIC if inputs are not stochastic. Additionally, the results of numerical experiments agree with our theoretical results