Self-clustering recurrent networks

It is shown, based on empirical analyses, that second-order recurrent neural networks which are trained to learn finite state automata (FSAs) tend to form discrete clusters as the state representation in the hidden unit activation space. This observation is used to define self-clustering networks which automatically extract discrete state machines from the learning network. To address the problem of instability a network structure is introduced, whereby the network uses quantization in the feedback path to force the learning of discrete states. Experimental results show that the method learns FSAs just as well as existing methods in the literature but with the significant advantage of being stable on test strings of arbitrary length