Automatic face modeling from monocular image sequences using modified non parametric regression and an affine camera model

We present the theory of modified nonparametric regression for estimating the 3D face structure of a human from a monocular image sequence. In the preprocessing stage, the face region is segmented from the background using both color and motion information, by using a hierarchical block motion estimation method. By using the affine camera projection geometry, and a given choice of an image frame pair in the sequence, we adopt the KvD model to express the depth at each point on the face region as a function of the unknown out-of-plane rotation, and some measurable quantities computed directly from the optical flow. This is repeated for multiple image pairs (keeping one fixed image frame which we formally call the “base” image, and choosing another frame from the sequence). The true depth map is next estimated from these equations using a modified nonparametric regression technique, and this forms the core contribution of this paper. We conducted experiments on various image sequences to verify the effectiveness of the technique, and propose to extend it for photorealistic modeling of arbitrary (non-face) objects from image sequences