Saturation in PSO neural network training: Good or evil?

Particle swarm optimisation has been successfully applied as a neural network training algorithm before, often outperforming traditional gradient-based approaches. However, recent studies have shown that particle swarm optimisation does not scale very well, and performs poorly on high-dimensional neural network architectures. This paper hypothesises that hidden layer saturation is a significant factor contributing to the poor training performance of the particle swarms, hindering good performance on neural networks regardless of the architecture size. A selection of classification problems is used to test this hypothesis. It is discovered that although a certain degree of saturation is necessary for successful training, higher degrees of saturation ultimately lead to poor generalisation. Possible factors leading to saturation are suggested, and means of alleviating saturation in particle swarms through weight initialisation range, maximum velocity, and search space boundaries are analysed. This paper is intended as a preface to a more in-depth study of the problem of saturation in particle swarm optimisation as a neural network training algorithm.