Modification of Backpropagation Algorithm and Its Application for Neural Networks with Threshold Activation Function

The method for determination of gradient of quadratic quality index of multi-layer neural network (MLNN) in one forward passage is proposed. Here, dependence of the gradient on the derivatives of the activation functions (AF) shall become obvious. Replacing the derivatives by the linearization coefficients of the activation functions shall make it possible to determine the coefficients of linearization of the quadratic quality index and to use these coefficients for determination of new values of the synaptic matrices in the supervisory learning procedure. As a result, extension of Backpropagation Algorithm (BPA) application to the networks with nondifferentiable and even discontinuous activation functions shall become possible. As an example, simple algorithm is proposed for determining the coefficients of linearization of the threshold-type activation function.