Error surfaces for multi-layer perceptrons

Explores some important characteristics of error surfaces for multilayer perceptrons. These characteristics help to explain why learning techniques which use hill climbing methods are so slow, and they provide insights into techniques that may help to speed up learning. Important characteristics revealed include the stair-step appearance of the surface, flat regions which extend to infinity in all directions, and a long narrow trough which leads to the minimum. In addition, the authors discuss the relationship between individual training samples and the steps of the surface, the effect of increasing the number of training samples, and the advantages of initializing the network weights to small random values