Title :
Singularities in neural networks make Bayes generalization errors smaller even if they do not contain the true
Author :
Watanabe, Sumio ; Amari, Shun-Ichi
Author_Institution :
Precision & Intelligence Lab., Tokyo Inst. of Technol., Yokohama, Japan
Abstract :
Hierarchical learning machines used in information science have singularities in their parameter spaces. At singularities, the Fisher information matrix becomes degenerate, resulting that the learning theory of regular statistical models can not be applied. Recently, it was proven that, if the true parameter is contained in the singularities, then the generalization error in Bayes estimation is far smaller than those of regular statistical models. In this paper, under the condition that the true parameter is not contained in singularities, we show two results: (1) if the dimension of the parameter from inputs to hidden units is not larger than three, then there exits a region of true parameters such that the generalization error is larger than those of regular models; and (2) if the dimension is larger than three, then, for an arbitrary true parameter, the generalization error is smaller than those of regular models.
Keywords :
Bayes methods; error statistics; estimation theory; generalisation (artificial intelligence); learning (artificial intelligence); neural nets; Bayes errors; Bayes estimation; Fisher information matrix; generalization error; hierarchical learning machines; neural networks; singularities; statistical models; Biological neural networks; Geometry; Information science; Intelligent networks; Laboratories; Machine learning; Multi-layer neural network; Neural networks; Neuroscience; Space technology;
Conference_Titel :
Neural Information Processing, 2002. ICONIP '02. Proceedings of the 9th International Conference on
Print_ISBN :
981-04-7524-1
DOI :
10.1109/ICONIP.2002.1202838