SAM: a perceptive spoken language-understanding robot

Speech activated manipulator (SAM), a reasoning robotic system with sensory capabilities that interacts with a human partner using natural language, is described. The robot understands, in real time, about 1041 semantically meaningful naturally spoken English language sentences using a vocabulary of about 200 words. SAM includes developments in mechanical control, real-time operating systems, multiprocessor communication and synchronization, kinematics, sensors and perception techniques, speech recognition and natural language understanding, robotic reasoning with gripper and arm motion planning. Speech recognition is augmented with semantic analysis and evaluation to form a complete speech understanding system. Used in conjunction with error recovery rules in the robot expert and frame-based knowledge system, SAM is robust and resistant to user errors. The most interesting aspects of the SAM system are described. Observations and experiences are discussed along with some advice for those interested in building similar systems