Programming manipulation tasks by demonstration in visuo-haptic augmented reality

This work investigates the use of visuo-haptic augmented reality for programming object manipulation tasks by demonstration. A desktop augmented reality system is presented in which the user is not co-located with the environment and a 3DOF haptic device, providing force feedback, is adopted for user interaction. The interaction paradigm allows the user to select and manipulate virtual objects superimposed upon a visual representation of the real workspace. The approach also supports physics-based animation of rigid bodies. In the proposed method automatic object recognition and registration are performed from 3D range data acquired by a laser scanner mounted on a robot arm. We next investigate the potential of the augmented reality system for programming manipulation tasks by demonstration. Experiments show that a precedence graph, encoding the sequential structure of the task, can be successfully extracted from multiple user demonstrations.