The TACOMA learning architecture for reflective growing of neural networks

One of the important problems to be solved for neural network applications is to find a suitable network structure solving the given task. To reduce the engineering efforts for the architecture design a data driven algorithm is desirable which constructs a network structure during the learning process. There are different approaches for structure adaptation with evolutionary algorithms, growth algorithms and others. To solve large problems successfully it is necessary to divide the problem into subproblems and to solve them separately by experts. This is a fundamental principle of nature. To implement this principle in artificial neural networks there are different approaches, but these algorithms yield fixed network structures. The authors propose a learning architecture for growing complex artificial neural networks which tries to include both sides of the coin, structure adaptation and task decomposition. The growing process is controlled by self-observation or reflexion. The algorithm generates a feedforward network bottom up by cyclically inserting cascaded hidden layers. Inputs of a hidden layer unit are locally restricted with respect to the input space by using a new kind of activation function, combining the local characteristics of radial basis function units with sigmoid units. Contrary to the cascade-correlation learning architecture the authors introduce different correlation measures to train the network units featuring different goals. The task decomposition between subnetworks is done by maximizing the anticorrelation between the hidden layer units output and a connection routing algorithm which only connects cooperative units of different layers. These features resemble the TACOMA (TAsk decomposition, COrrelation Measures and local Attention neurons) learning architecture. Self-observation is done by transforming the errors and the network structure to the input space. So it is possible to infer from errors to structure and reverse