Incremental learning using the time delay neural network

The time delay neural network (TDNN) is one of the neural network architectures that give excellent performance in tasks involving classification of temporal signals, such as phoneme classification, on-line gesture and handwriting recognition, and many others. One particular problem that occurs in on-line recognition tasks is how to deal with input patterns that are incorrectly recognized because they are totally dissimilar to anything the network has seen during training. The author presents an algorithm to add incremental, one-shot learning capability to the TDNN by creating extra hidden units to perform template matching on incorrectly recognized inputs and influence the output units via excitatory or inhibitory connections. In a simple handwritten digit recognition task, the addition of a single extra unit increases recognition rate for a new digit variation from 0% to 99%, while decreasing the performance on the old data by only 0.6%. Thus this incremental TDNN (ITDNN) can in fact learn a new pattern from one example and perform reasonably well on similar inputs without forgetting what it already knew, thereby enabling it to deal effectively with the on-line misrecognition problem