Mobile manipulator control based on voice and visual signal

In this paper, two kinds of methods are used to control the mobile manipulator in terms of voice signal control approach and visual signal control approach. In voice signal control approach, the mobile manipulator can perform some actions according to the operator´s speaking voice input through a microphone. For example, the robot can be commanded to go forward or backward in specified distance, turn left or right with specified angle, even can follow up “8” shape trajectory and catch a ball. In visual approach, the mobile robot can accomplish some actions according to the operator´s gestures of two hands using Kinect sensor information. It gets the depth image of two hands in spatial coordinates and calculates the relative distance of them to take some actions by an onboard computer. An embedded single board computer is installed in mobile robot Pioneer2 DX for enhancing remote control and running program´s ability.