Multimedia fusion in automatic extraction of studio speech segments for spoken document retrieval

The paper describes our progress in Cantonese spoken document retrieval. Over 60 hours of Cantonese television news broadcasts have been collected as part of the AoE-IT Multimedia Repository. We have also developed the multimedia markup language (MmML) for annotating the multimedia content in terms of anchor/field video frames and audio recordings. The audio tracks are indexed by a Cantonese syllable recognizer. Our investigation indicates that there is a large discrepancy in recognition performance, i.e. dropping from 59% to 39% in syllable accuracy (and corresponding reliability in audio indexing), as we move from anchor speech recorded in the studio to reporter/interview speech recorded in the field. Hence we present several automatic methods to extract anchor/studio speech from the audio tracks for retrieval: (i) extraction based only on video information using a fuzzy c-means algorithm; (ii) extraction based only on audio information using Gaussian mixture models; (iii) a fusion strategy that combines video- and audio-based extraction. The paper presents the performance of various extraction techniques and the related retrieval performance in a known-item spoken document retrieval task.