Human-robot interaction with multi-sensor fusion based hand sign recognition for service robot

In this paper we introduce a combining method with multi-sensor fusion for hand sign recognition and apply this method on service robot. Hand sign recognition is an essential way for Human-Robot Interaction (HRI). Sign language is the most intuitive and direct way to communicate with impaired or disabled people. Through the hand or body gestures, the disabled can efficiently let caregiver or robot know what message and instruction they want to convey. In this paper, we propose a combined hands gesture recognition algorithm which combines two distinct recognizers. These two recognizers collectively determine the hand´s sign via a process called combinatorial approach recognizer (CAR) equation. These two recognizers are aimed to complement the ability of discrimination. To achieve this goal, one recognizer recognizes hand gesture by hand skeleton (HSR), and the other recognizer is based on support vector machines (SVM). In addition, the corresponding classifiers of SVM are trained using different features which are Gabor feature and raw data. Furthermore, the trained images are using Bosphorus Hand Database and in addition to taking by us. A set of rules including recognizer switching and combinatorial approach recognizer (CAR) equation is devised to synthesize the distinctive methods. To achieve the human hand sign recognition in the correct statement, we equip two sensors on the service robot and implement the above algorithms on it. These two sensors are CCD sensor and laser range finder (LRF). We have successfully demonstrated signs recognition experimentally with quite satisfactory proof of concept results.