A diffusion process for global optimization in neural networks

The authors modify the usual gradient descent method to push the process in the weight space to have a Gibb or Boltzmann distribution, and find the global minima of the average performance measure of a neural network. The goal is to present a method which guarantees that a global minima of the average performance measure in the weight space will be located, given sufficient computational time. The method of simulated annealing is a mathematical tool which forces a system to behave like a natural annealing process. The method chosen for the global optimization of continuous networks is based on the modification of the differential equation associated with local optimization. The global optimization theory is derived for networks whose learning rules are supervised, whose nodes are bounded Lipschitz continuous functions, and whose performance measure is smooth