Bifurcations in nonlinear networks initialized with linear network weight solutions

It has been reported (Yang and Wu, 1993) that weight convergence of multilayer perceptron networks (MLP) can be improved by smoothly changing the node nonlinearity from a linear to a sigmoidal function during training. While this approach might provide a useful training heuristic, formally, this method depends on an underlying homotopy which transforms a linear to a nonlinear network of the same architecture. As a parameter τ (the homotopy parameter) is varied from ten, to one, the linear network weights are mapped onto nonlinear network weight solutions. In this paper, a geometric interpretation of the optimization equations is used to construct and describe an example network that illustrates practical and theoretical difficulties resulting due to bifurcation of solution paths. Since the linear system is a generic high-order bifurcation point of the homotopy equations, solution paths are discontinuous at initialization. Bifurcations and infinite solutions for intermediate values τ∈(0,1) also can occur for data sets which are not of measure zero. These results weaken the guarantees on global convergence and exhaustive behavior normally associated with homotopy methods. The geometric perspective further provides insight into the relationship between linear and nonlinear perceptron networks, and how weight solutions arise in each