Gesture recognition using video and floor pressure data

This paper presents a multimodal gesture recognition framework using video and floor pressure data. The key contribution of this research is to show that using additional floor pressure data significantly improves the recognition of visually ambiguous gestures. To effectively combine gesture recognition results from both the visual and pressure sensing modalities, we have adopted a two-stage cascaded sequential information integration scheme. In Stage-1 of the scheme, an unknown movement segment is first classified into a gesture group based on the visual features, and then in Stage-2, the input movement is further recognized as a gesture within the gesture group according to the pressure features. In the proposed framework, the hidden Markov models (HMMs) are used to model and recognize gestures using features from video and pressure data. The experimental results obtained on an in-house video and floor pressure gesture dataset demonstrate the efficacy of the proposed multimodal gesture recognition framework.