Weight decay and resolution effects in feedforward artificial neural networks

Results are presented from a preliminary study on the effects of weight decay and resolution on the performance of typical three-layer, feedforward neural networks. Two types of decay are investigated, unilateral decay toward the most negative weight (unipolar) and bilateral decay toward the median or zero weight value (bipolar), and compared with Gaussian weight perturbations. This analysis is pertinent to the area of VLSI-based network implementations with analog weight storage. The results show that, if weight decay is unavoidable, bipolar decay achieves an order-of-magnitude better performance than unipolar and that the weight resolution required in actual implementations of feedforward, connectionist hardware is higher than predicted by computer simulations of network responses to random or Gaussian weight perturbations