Variable Projection Method and Levenberg-Marquardt Algorithm for Neural Network Training

Author

Kim, Cheol-Taek ; Lee, Ju-Jang ; Kim, Hyejin

Author_Institution

KAIST, Daejeon

fYear

2006

fDate

6-10 Nov. 2006

Firstpage

4492

Lastpage

4497

Abstract

An optimal solution for single hidden layered feedforward neural network (SLFN) is proposed. SLFN can be considered as separable nonlinear least squares problem and variable projection (VP) method gives the approximation of the Jacobian matrix of the problem. The Jacobian calculation of VP-SLFN is suggested with simplified form. Based on this simplification, Levenberg-Marquardt (LM) algorithm for VP-SLFN is suggested and has faster convergence rate than LM algorithm without VP. Two numerical examples show the superiority than extreme learning machine and LM without variable projection

Keywords

Jacobian matrices; feedforward neural nets; learning (artificial intelligence); least squares approximations; Jacobian matrix; Levenberg-Marquardt algorithm; learning machine; neural network training; separable nonlinear least squares problem; single hidden layered feedforward neural network; variable projection method; Feedforward neural networks; Function approximation; Iterative algorithms; Jacobian matrices; Least squares approximation; Least squares methods; Machine learning; Neural networks; Newton method; Recursive estimation;

fLanguage

English

Publisher

ieee

Conference_Titel

IEEE Industrial Electronics, IECON 2006 - 32nd Annual Conference on

Conference_Location

Paris

ISSN

1553-572X

Print_ISBN

1-4244-0390-1

Type

conf

DOI

10.1109/IECON.2006.347644

Filename

4153184