A Hierarchical Framework For High Resolution Facial Expression Tracking

We present a novel hierarchical framework for high resolution, nonrigid facial expression tracking. The high quality dense point clouds of facial geometry moving at video speeds are acquired using a phase-shifting based structured light ranging technique. To use such data for temporal study of the subtle dynamics in expressions and for face recognition, an efficient nonrigid facial tracking algorithm is needed to establish intra-frame correspondences. In this paper, we propose such an algorithmic framework that uses a multi-resolution 3D deformable face model, and a hierarchical tracking scheme. This framework can not only track global facial motion that is caused by muscle action, but fit to subtler expression details that are generated by highly local skin deformations. Tracking of global deformations is performed efficiently on the coarse level of our face model with one thousand nodes, to recover the changes in a few intuitive parameters that control the motion of several deformable regions. In order to capture the complementary highly local deformations, we use a variational algorithm for non-rigid shape registration based on the integration of an implicit shape representation and the Free Form Deformations (FFD). Due to the strong implicit and explicit smoothness constraints imposed by the algorithm, the resulting registration/deformation field is smooth, continuous and gives dense one-to-one intra-frame correspondences. User-input sparse facial feature correspondences can also be incorporated as hard constraints in the optimization process, in order to guarantee high accuracy of the established correspondences. Extensive tracking experiments using the dynamic facial scan of five different subjects demonstrate the accuracy and efficiency of our proposed framework.