A robust sketch interface for natural robot control

A fully probabilistic command interface for controlling robots using multi-stroke sketch commands is presented. Drawing from prior work in handwriting recognition, sketches are modeled as a variable duration hidden Markov model, where the distributions on the states and transitions are learned from training data. A forward search algorithm on the gesture, stroke, and stroke transition observations is used to find the most likely sketch, which is displayed to the user for confirmation. In cases where the most likely sketch is incorrect, the user can reject it, prompting the next most likely sketch to be displayed. Upon confirmation from the user, the robot executes the desired behaviors. A prototype sketch interface was implemented using a pen tablet; two sets of search-and-identify experiments were conducted using a single robot in an indoor environment to test the usability of the proposed framework. Even novice users were able to successfully complete the missions, including those on whom the algorithm was not trained. User surveys indicate that operators generally found the interface to be natural and easy to use.