Application of markerless image-based arm tracking to robot-manipulator teleoperation

In robot teleoperation, contacting mechanical devices and sensors have been commonly used to track operator hand and arm motion. While camera-based tracking has the benefit of being non-contacting, markerless camera-based human tracking offers a further advantage of not requiring markers and thus avoiding marker occlusion. This paper presents an application of markerless image-based arm tracking to real-time teleoperation of a robot manipulator. The markerless tracking is carried out by processing images from two calibrated cameras in real-time, to estimate the positions of the joint centres of the wrist and elbow in three dimensions (3D), and to compute the 3D positions of the index finger and thumb in order to estimate the hand orientation. These are used to determine the position and orientation of the endeffector of a robot-manipulator in real-time teleoperation. Markerless tracking for teleoperation was demonstrated for pick-and-place tasks.