Camera and microphone array for 3D audiovisual face data collection

This paper proposes a novel camera/microphone array system capable of capturing dynamic facial expression video with synchronized speech and reconstructing realistic 3D face models from the data. Both hardware and software issues including camera calibration, video/audio synchronization, facial marker tracking and 3D shape reconstruction are considered. To our best knowledge, this system is the first camera/microphone array system that is able to capture high-resolution facial expression video with synchronized speech. The system can be used to collect dynamic 3D audiovisual face data for many multimedia applications.