Combining online and offline learning for tracking a talking face in video

Facial appearance changes in video sequences represent a non stationary data problem, because of factors such as variations in pose, illumination and facial expressions. While most algorithm, that employ fixed appearance models of the target object, are not robust to track objects in uncontrolled environments. Existing Adaptive Appearance Models (AAMs) approaches solve this problem to an extent. However, they are not able to detect a misalignment, partial occlusions and do not adequately track facial feature points such as those relating to the eyes or mouth in the presence of significant expression changes. In this paper, we propose to combine an online and an offline learning approaches for robust tracking of feature points of a talking face. The online learning used in a stochastic approach to estimate facial feature points globally and in a deterministic approach to refine the feature points. The tracked results are filtered by offline learning approach to ensure rejection of poorly aligned targets. This allows the proposed tracker to significantly improves robustness against appearance changes and occlusions. Experiment results on tracking facial feature points in long video sequences with a wide range of facial expressions in head movement demonstrate the effectiveness and robustness of our tracker.