Continuous speech recognition using segmental neural nets

The authors present the concept of a `segmental neural net´ (SNN) for phonetic modeling in continuous speech recognition (CSR) and show how this can be used, together with a hidden Markov model (HMM) system, to improve continuous speech recognition (CSR). The SNN is a segment-based model that uses a neural network to correlate features of the speech signal throughout the duration of a phonetic segment. The problem of handling phonetic segments of varying length is solved by applying a warping function which provides the neural network inputs with a fixed-length representation of the segment. This method of modeling speech differs from that of HMMs, which assume that speech frames are conditionally independent. To take advantage of the training and decoding speed of HMMs, a hybrid SNN/HMM system has been developed to combine the advantages of both types of approaches. In this hybrid system, use is made of the N-best paradigm to generate likely phonetic segmentations, which are then scored by the SNN. The HMM and SNN scores are then combined to optimize performance