Recovering human pose in 3D by visual manifolds

We describe a mechanism based upon activity manifolds that map image data from more than one view to spatial pose. We learn the manifolds from training data which are motion capture data about real human subjects exercising the target actions. The nature of the training data allows the learned manifolds to conform naturally to multiple constraints, including (1) the body-part articulation constraint; (2) the image-consistency constraint; and (3) conformation to prior information about the possible human activities. A mirror function is specifically designed that allows the system to pick up the proper manifold in multi-activity scenario. Human pose in both the image space and 3D is expressed in terms of the body-part joint positions. Such a representation allows image data to be related across views and to 3D space with ease. Experimental results show that not only do the manifolds effectively map image data to 3D pose; the presence of multiple images also improves the precision of the recovered pose and helps fix feature extraction error in any single image.