A humanoid robot that listens, speaks, sees and manipulates in human environments

Robots providing services in human environments will need to be endowed with the capabilities of listening, speaking, seeing and manipulating. This paper presents an integrated humanoid robot system that is able to perform tasks in human environments responding to spoken language commands, locate objects using vision function and manipulate objects by generating whole body motions on-line. A system that is able to understand natural language instructions and act accordingly will need the integration of knowledge representation, perception, decision making and on-line motion generation technologies. This paper reports on our ongoing endeavor to create a reliable and flexible integrated system that enables a human user to operate humanoid robots using two levels of continuous motion level operation and discrete behavior level operation through natural language instructions. We report on the implementation of the proposed system on humanoid robot HRP-2, which is able to locate auditory sources and receive natural language instructions from a user within 2 meters using a 8-channel microphone array connected to a speech recognition embedded system on-board the robot.