On sparsity-exploiting memory-efficient trust-region regularized nonlinear least squares algorithms for neural-network learning

This paper highlights nonlinear least squares algorithms with trust-region regularization for multiple-output neural-network (NN) models, describing how special structures of the "block-angular" residual Jacobian matrix and the "block-arrow" Gauss-Newton Hessian (or Fisher information matrix) can be exploited to render a large class of NN-learning algorithms "efficient" in both memory and operation counts. In simulation, we demonstrate both direct and iterative trust-region algorithms with two distinct nonlinear models: "multilayer perceptrons (MLP)" and "complementary mixtures of NN-experts" (or neuro-fuzzy modular networks) using a relatively large real-world nonlinear regression application.