Multi-source face tracking with audio and visual data

A real-time face tracker based on both sound and visual cues is presented. Initial talker locations are estimated acoustically from microphone array data while precise localization and tracking are derived from visual data. The image processing employs a hierarchical structure which utilizes source motion, contour geometry, color data, and facial features. The resulting system is capable of tracking multiple persons in complex backgrounds and robustly discriminating faces from similar objects. While the direct focus of this work is automated videoconferencing, the face tracking capability has utility to many multimedia and virtual reality applications