Robust Real-Time Face Pose and Facial Expression Recovery

Face motion is the sum of rigid motion related with face pose and non-rigid motion related with facial expression. Both motions are coupled in the captured image so that they can not be easily recovered from the image directly. In this paper, a novel technique is proposed to recover 3D face pose and facial expression simultaneously from a monocular video sequence in real time. First, twenty-eight salient facial features are detected and tracked robustly under various face orientations and facial expressions. Second, after modelling the coupling between face pose and facial expression in the 2D image as a nonlinear function, a normalized SVD (N-SVD) decomposition technique is proposed to recover the pose and expression parameters analytically. A nonlinear technique is subsequently utilized to refine the solution obtained from the N-SVD technique by imposing the orthonormality constraint on the pose parameters. Compared to the original SVD technique proposed in [1], which is very sensitive to the image noise and numerically unstable in practice, the proposed method can recover the face pose and facial expression robustly and accurately. Finally, the performance of the proposed technique is evaluated in the experiments using both synthetic and real image sequences.