Feedback-based handwriting recognition from inertial sensor data for wearable devices

This paper presents a novel interactive method for recognizing handwritten words, using the inertial sensor data available on smart watches. The goal is to allow the user to write with a finger, and use the smart watch sensor signals to infer what the user has written. Past work has exploited the similarity of handwriting recognition to speech recognition in order to deploy HMM based methods. In contrast to speech recognition, however, in our scenario, the user can see the individual letters that are recognized on a sequential basis, and provide feedback or corrections after each letter. In this paper, we exploit this key difference to improve the input mechanism over a classical source-channel model. For a small increase in the amount of time required to input a word, we improve recognition accuracy from 59.6% to 91.4% with an implicit feedback mechanism, and to 100% with an explicit feedback mechanism.