Pen Acoustic Emissions for Text and Gesture Recognition

The sounds generated by a writing instrument provide a rich and under-utilized source of information for pattern recognition. We examine the feasibility of recognition of handwritten cursive text, exclusively through an analysis of acoustic emissions. Our recognizer uses a template matching approach, with templates and similarity measures derived variously from: raw power signal with fixed resolution, discrete sequence of magnitudes obtained from peaks in the power signal, and ordered tree obtained from a scale space signal representation. Test results are presented for isolated lowercase cursive characters and for whole words. Recognition rates of over 70% (alphabet) and 90% (26 words) are achieved, based solely on acoustic emissions, with samples provided by a single writer. We also present qualitative results for recognizing gestures such as circling, scratch-out, check-marks, and hatching. These preliminary results demonstrate that acoustic emissions are a rich source of information, usable - on their own or in conjunction with image-based featuresi - to solve pattern recognition problems. In future work, this approach can be used in applications such as writer identification, handwriting and gesture-based computer input technology, emotion recognition, and temporal analysis of sketches.