Bi-manual robotic paper manipulation based on real-time marker tracking and physical modelling

The ability to manipulate deformable objects, such as textiles or paper, is a major prerequisite to bringing the capabilities of articulated robot hands closer to the level of manual intelligence exhibited by humans. We concentrate on the manipulation of paper, which affords us a rich interaction domain and that has not yet been solved for anthropomorphic robot hands. A key ability needed for this is the robust tracking and modelling of paper under conditions of occlusion and strong deformation. We present a marker based framework that realizes these properties robustly and in real-time. We compare a purely mathematical representation of the paper manifold with a soft-body-physics model and demonstrate the use of our visual tracking method to facilitate the coordination of two anthropomorphic 20 DOF Shadow Dexterous Hands while they grasp a flat-lying piece of paper, using a combination of visually guided bulging and pinching.