Incremental communication for multilayer neural networks

A new method of inter-neuron communication called incremental communication is presented. In the incremental communication method, instead of communicating the whole value of a variable, only the increment or decrement of its previous value is sent on a communication link. The incremental value may be either a fixed-point or a floating-point value. Multilayer feedforward network architecture is used to illustrate the effectiveness of the proposed communication scheme. The method is applied to three different learning problems and the effect of the precision of incremental input-output values of the neurons on the convergence behavior is examined. It is shown through simulation that for some problems even four-bit precision in fixed- and/or floating-point representations is sufficient for the network to converge. With 8-12 bit precisions almost the same results are obtained as that with the conventional communication using 32-bit precision. The proposed method of communication can lead to significant savings in the intercommunication cost for implementations of artificial neural networks on parallel computers as well as the interconnection cost of direct hardware realizations. The method can be incorporated into most of the current learning algorithms in which inter-neuron communications are required. Moreover, it can be used along with the other limited precision strategies for representation of variables suggested in literature