Smooth Adaptive Fitting of 3D Face Model for the Estimation of Rigid and Non-rigid Facial Motion in Video Sequences

In this paper, we propose a two-level integrated model for accurate 3D tracking of rigid head motion and non-rigid facial animation. At the lowest level, the 2D shape of facial features is robustly extracted using a regularized shape model and a cascade multi-stage algorithm. At the highest level, we estimate both the facial animation and 3D pose parameters via minimizing an energy function comprising three terms. The first quantifies the error in matching a 3D wireframe face model (Candide) to the image sequence, while the remaining terms impose temporal and spatial motion-smoothness constraints over the 3D model points. Through extensive experiments, we demonstrate the feasibility and effectiveness of the proposed method and also show it outperforms the state-of-the-art algorithms such as the Bayes Tangent Shape Model.