GPU implementation of the feedforward neural network with modified Levenberg-Marquardt algorithm

In this paper, an improved Levenberg-Marquardt-based feedforward neural network, with variable weight decay, is suggested. Furthermore, parallel implementation of the network on graphics processing unit is presented. Parallelization of the network is achieved on two different levels. First level of parallelism is data set level, where parallelization is possible due to inherently parallel structure of the feedforward neural networks. Second level of parallelism is Jacobian computation level. Third level of parallelism, i.e. parallelization of optimization search steps, is not implemented due to the variable weight decay, which makes third level of parallelism redundant. Suggested weight decay variation enables the compromise between higher accuracy with oscillations on one side and stable, but slower convergence on the other. To improve learning speed and efficiency, modification of random weight initialization is included. Testing of proposed algorithm is performed on two real domain benchmark problems. The results obtained and presented in this paper show effectiveness of proposed algorithm implementation.