Multimodal English corpus for automatic speech recognition

A multimodal corpus developed for research of speech recognition based on audio-visual data is presented. Besides usual video and sound excerpts, the prepared database contains also thermovision images and depth maps. All streams were recorded simultaneously, therefore the corpus enables to examine the importance of the information provided by different modalities. Based on the recordings, it is also possible to develop a speech recognition system which analyzes many modalities at the same time. The paper describes the process of multimodal material collection and the post-processing procedure applied to this material. Parameterization methods of signals belonging to different modalities are also proposed.