Real-time 3-D face tracking and modeling framework for mid-res cam

We present a robust, real-time 3-D face tracking and modeling system providing accurate 6 degree-of-freedom head pose in the presence of large out-of-plane motion, strong expression changes, and partial occlusions. In this paper, we have extended the previous 3-D face tracking and modeling framework [10] with automatic initialization, reacquisition, and automatic pose correction. Our system first generates a 3-D face model from a single frontal image. We then extract uniformly distributed random points and track them in 2-D. Given these correspondences, the 3-D head pose is robustly estimated using a RANSAC-PnP process. As the head moves, we dynamically add new feature points to handle a large range of poses. A measure of the accumulated error over time allows an auto-correction mechanism to recover from drift when necessary. If the tracker gets lost, due to motion blur or strong occlusions, the system re-initializes. We present live demo results, which shows excellent tracking under large motion (roll: 360°, yaw: ±90°, pitch: -60° to +90°), fast movement, occlusion and facial expression variations. The system runs at 14 fps on a laptop CPU. By experiments on different datasets, our method shows state of the art results.