Unified multiple media interface for robot teleoperation

The power of multimedia in communicating with lay users has not been fully exploited in controlling real devices (C.J. Weinstien, 1995; 1991). The authors develop a unified interface using multiple media modalities (speech, gesture (image) and video) for controlling a diverse set of devices. The same instruction effects different behaviors in different contexts (e.g. “left” to a robot arm causes a different motion than “left” to a camera). This captures the context sensitivity in natural communication, which overloads symbols with multiple meanings applicable in different contexts. This also provides for an approach to unify several interfaces into one by using context sensitivity as a tool for disambiguation. Results are shown for implementations based on a common unified direct telephone interface for speech, and a single camera visual interface for gestures for three different devices. Audio and video are also used as a feedback for naturalistic interaction between the operator and the device