Precision analysis of stochastic pulse encoding algorithms for neural networks

The authors present mathematical results related to recent neural network algorithms employing stochastic pulse encoding. In such algorithms, neural activations and connection weights are encoded as stochastic streams of pulses, where the average density represents the signal or weight value. The authors show the precise form of expected output for two- and three-input neurons, and describe these functions in the limit for a large number of inputs. They address a fundamental limitation inherent in these stochastic techniques: their finite precision. The precision is dependent upon the pulse averaging period-the longer this period (i.e. the larger the number of pulses sampled), the higher the precision. The authors derived exact expressions for the distribution of neural periods as well as a statistical analysis to find the averaging period required for precision of five bits-a resolution determined by others to be necessary for successful implementations of backpropagation. It is found that ≈1000 pulses are required for 5-b precision. These results reveal fundamental limits in speed and memory requirements for stochastic pulse implementations of neural learning algorithms