Robust Pose Invariant Facial Feature Detection and Tracking in Real-Time

In this paper, a robust technique is proposed to detect and track a set of twenty-eight prominent facial features under various facial expressions and face orientations in real-time. Specifically, after the face image is captured from the camera, a trained face mesh is first employed to estimate a rough position for each facial feature based on the located eye positions. Subsequently, an accurate position is obtained for each facial feature by searching around its roughly estimated position. Once the facial features are located, by using the appearance information of each facial feature together with the geometry information among the facial features, a shape-constrained correction-based tracking mechanism is activated to track them in the subsequent image frames. Finally, the performance of the proposed technique is demonstrated through building a real-time facial feature tracking system that can detect and track a set of twenty-eight facial features automatically as soon as a person is sitting in front of the camera