Creating, composing, and recognizing multisensor gestures in mobile devices

The work reported here focuses on creating, composing, and recognizing multisensor gestures and their use in terms of remotely controlling another device. The approach allows to compose gestures synthetically, i.e., by recording complex gestures directly as they are performed, and analytically, i.e., by selecting and composing existing gestures to define more complex ones. Analytical gesture composition doesn´t have to be sequential: the gestures can be superposed. We explored this in a setting of using a mobile device to control a remote device. In our case, this was a smartphone controlling a computer, which we assume to be most probable setting in practice. For this, a client-server system was developed consisting of the application running on the Android mobile device using the accelerometer, proximity sensor, touchscreen, and ambient light sensor to capture gestures. The server-side application that runs on a computer was developed to store and recognize gestures. The preliminary evaluation results are promising exhibiting the 85% rate in the ability of users to replicate gestures not created by them. According to informal observations, the user seem to be quite comfortable working with their own multisensor gestures.