A double-talk-detector using sound and image information

In this paper, we propose a double-talk-detector using multi-modal information (sound and image). An acoustic echo cancellation is used for hands-free telecommunication and teleconference systems. However, the performance of the acoustic echo cancellation deteriorates according to a double talk where the near-end talker and the far-end talker simultaneously utter. For this problem, the acoustic echo canceller (AEC) using Sub-Adaptive-Filter (Sub-ADF) has been already proposed. However, the double-talk detector cannot detect double-talk situations correctly. Therefore, we propose a double-talk detector using multi-modal information in order to improve the performance of the double-talk detector. The proposed double-talk detector detects a voice activity from image information which is obtained from binarized lip image and acoustic information which is obtained from the correlation between the microphone output and the adaptive filter output. Simulation results demonstrate that the proposed double-talk detector can improve the performance compared with the conventional one.