Online speech-reading system for Japanese language

In this paper, a speech-reading system dedicated for Japanese language is introduced. The Japanese language has unique feature that its letters must end with a English vowel character (a, i, u, e, and o). Using this feature, it is possible to quantize the continuous lip motion into a discrete sequence of vowels, which can classify the spoken sentences. This technique can achieve high accuracy especially when the required sentences to recognize is defined in advance. In this paper, the quantization process is performed using the Hypercolumn neural network model (HCM), which consists of hierarchical layers of the Hierarchical Self-Organizing Maps (HSOM) neural network arranged as the cell planes of the Neocognitron (NC) neural network. HCM can recognize images with variant objects size, position, and spatial resolution. However, due to the hierarchical structure of the HCM model, the network spends long time in the recognition. To achieve on-line recognition, during the recognition phase a new competition algorithm for the HCM is proposed. This algorithm is based on selecting subset from the network codebook, which can reduce the network recognition time into the range of real-time rate. Results show that the system perform well in the online recognition of eight Japanese sentences.