Optimally generalizing neural networks

The problem of approximating a real function f of L variables, given only in terms of its values y₁,. . .,y_M at a small set of sample points x₁,. . .,x_M in R^L, is studied in the context of multilayer neural networks. Using the theory of reproducing kernels of Hilbert spaces, it is shown that this problem is the inverse of a linear model relating the values y_m to the function f itself. The authors consider the least-mean-square training criterion for nonlinear multilayer neural network architectures that learn the training set completely. The generalization property of a neural network is defined in terms of function reconstruction and the concept of the optimally generalizing neural network (OGNN) is proposed. It is a network that minimizes a criterion given in terms of the true error between the original function f and the reconstruction f₁ in the function space, instead of minimizing the error at the sample points only. As an example of the OGNN, a projection filter (PF) criterion is considered and the PFGNN is introduced. The network is of the two-layer nonlinear-linear type