Integrating vision, haptics and proprioception into a feedback controller for in-hand manipulation of unknown objects

We propose a feedback-based solution for the accurate manipulation of an unknown object in hand. This method does not explicitly models friction and surface geometry details, but employs a fast feedback loop based on visual and tactile feedback to perform robust manipulation even in the presence of unexpected slippage or rolling. At every control step, fingertip motions are computed to realize the intended object relocation, employing a composite position/force controller. Subsequently inverse hand kinematics is employed to retrieve joint-level motions, which are implemented on the robot with a position servo loop. We evaluate our method on a setup of two KUKA robot arms, each equipped with a tactile sensor array as end-effectors to perform the object manipulation task. The experimental results show the feasibility of our proposed method, even in presence of slippage or external disturbances.