A visuo-tactile control framework for manipulation and exploration of unknown objects

We present a novel hierarchical control framework that unifies our previous work on tactile-servoing with visual-servoing approaches to allow for robust manipulation and exploration of unknown objects, including - but not limited to - robust grasping, online grasp optimization, in-hand manipulation, and exploration of object surfaces. The framework is divided into three layers: a joint-level layer, a tactile servoing layer, and a visual servoing layer. While the middle layer provides "blind" surface exploration skills, maintaining desired contact patterns, the visual layer monitors and controls the actual object pose providing high-level fingertip motion commands that are merged with the tactile-servoing control commands. We illustrate the versatility of the proposed framework using a series of manipulation actions performed with two KUKA LWR arms equipped with a tactile sensor array as a "sensitive fingertip". The two considered objects are unknown to the robot, i.e. neither shape nor friction properties are available.