Kinect-based robotic manipulation: From human hand to end-effector

This paper presents a novel robot motion control method in real-time using the Kinect-based hand tracking. Making use of the middleware NiTE2 modules for Kinect, the 3D position of a hand center point can be captured steadily, which is applied to control the position of the robot´s end-effector. Meanwhile, an effective method is presented to estimate another two fingertip points in 3D space by processing hand depth images from Kinect. By utilizing these three points (a hand center and two fingertips) a coordinate system can be built up easily, which is used to specify the hand pose. In the same way, the hand pose is used to control the orientation of the end-effector. What´s more, the opening and closing actions of gripper are controlled by changing the distance between these two fingertips. This method enables an operator to control the end-effector easily with only one hand in actual real time. The effectiveness and efficiency of the presented method have been verified in the water-pouring tasks with a 5-DOFs modular manipulator.