Measuring Saturation in Neural Networks

In the neural network context, the phenomenon of saturation refers to the state in which a neuron predominantly outputs values close to the asymptotic ends of the bounded activation function. Saturation damages both the information capacity and the learning ability of a neural network. The degree of saturation is an important neural network characteristic that can be used to understand the behaviour of the network itself, as well as the learning algorithm employed. This paper suggests a measure of saturation for bounded activation functions. The suggested measure is independent of the activation function range, and allows for direct comparisons between different activation functions.