Person tracking via audio and video fusion

In this paper we present a joint audio-video (AV) tracker which can track the active source between two freely moving persons speaking in turn to simulate a meeting scenario, but less constrained. Our tracker differs from existing work in that it requires only a small number of sensors, works when speaker is not close to the sensors and relies on simple, yet efficient, inference techniques in AV processing. The system uses audio and video measures of the target position on the ground plane to strengthen the single modality predictions that would be weak if taken on their own as occlusions, clutter, reverberations and speech pauses happen in the test environment. In particular, the inter-microphone signal delays and the target image locations are input to single modality Bayesian filters, whose proposed likelihoods are multiplied in a Kalman Filter to give the joint AV final estimation. Despite the low complexity of the system, results show that the multi-modal tracker does not fail, tolerating video occlusion and intermittent speech (within 50 cm of accuracy) in the context of a non-meeting scenario. The system evaluation is done both on single modality than multi-modality tracking, and the performance improvement given by the AV fusion is discussed and quantified i.e 24 % improvement on the audio tracker accuracy.