Induced specialization of context units for temporal pattern recognition and reproduction

Additional inputs to a feedforward network, derived from the output of the hidden layer neurons, allow a feedforward network to deal with temporal pattern recognition and reproduction tasks. These `network derived´ or `context´ inputs augment the `true´ inputs to the network and allow the network to retain past information necessary for temporal sequence processing. The choice of which hidden neurons to retain to provide the context inputs is difficult. Use of all the hidden neurons increases the size of the overall network resulting in poorer generalization performance. The problem is complicated due to difficulty in choosing the number of hidden layer neurons in the first place. In this paper, we propose the use of regularization terms in the sum-of-squared error cost function. Assuming the hidden layer neurons are indexed 1,2,...,m, the regularization terms force the differentiation of hidden neurons 1 through m₁, and m₂ through m (where 1<m₁<m₂<m). Both m₁ and m₂ are controllable and allow fringe neurons to be used to provide the context inputs if the number of context units to use is known. When the number of context neurons to use cannot be determined, the regularization terms minimize m₁, and maximize m₂, while hidden neurons m₁ through m ₂ are penalized for differentiation. An amplitude detection simulation is used to evaluate the efficacy of the proposed paradigm