Robot guidance by human pointing gestures

In this paper we report on the development of the modular neural system “SEE-EAGLE” for the visual guidance of robot pick-and-place actions. Several neural networks are integrated to a single system that visually recognizes human hand pointing gestures from stereo pairs of color video images. The output of the hand recognition stage is further processed by a set of color-sensitive neural networks to determine the Cartesian location of the target object that is referenced by the pointing gesture. Finally, this information is used to guide a robot to pick the target object and place it at another location that can be specified by a second pointing gesture. The accuracy of the combined system allows one to identify the location of the referenced target object to an accuracy of 1 cm in a workspace area of 50×50 cm. In our current environment, this is sufficient to pick and place arbitrarily positioned target objects within the workspace