Combining acoustic and visual modalities in vowel recognition system for laryngectomees

This paper addresses the problem of vowels recognition in patients after total laryngectomy using combined visual and acoustic features. The linear prediction coefficients were estimated from speech signal using weighted recursive least squares algorithm. Ten cross-sectional areas of vocal tract model were calculated. Face expression parameters related to the spoken vowel were extracted from video recordings. Lips width, lips height and jaw opening were measured from grabbed video frames. The principal component analysis was applied to show correlations of auditory and visual features. The vowel recognition procedures were based on single hidden layer neural networks. The recognition performances of visual, acoustic and fused modalities were compared. It was presented that recognition performance of sustained vowels using 10 cross-sectional areas estimates is very low. Facial expression analysis is needed when there is problem with estimation of standard acoustic parameters of pathological speech.