An investigation into the use of nonlinear neural network architectures

Most research into neural nets has concentrated on either applications or learning algorithms. Comparatively little work has been done on studying the functional limitations imposed by the choice of network architecture. In this paper an investigation into the use of activation functions within the units of a network is presented. It is shown how such functions can affect the network behaviour, and why these changes take place. Efforts are particularly concentrated on the unit activation functions since they are likely to have a major impact on network performance and also since very little work has been carried out in this area. The results of this study lend confidence to the idea that considering alternative architectures may be useful