HMM-based Arabic Sign Language Recognition using Kinect

With research in Arabic Sign Language Recognition (ArSLR) still in its infancy, we present a method for recognition of Arabic Sign Language (ArSL) using Microsoft Kinect. A new dataset for ArSL words was collected using Kinect due to the lack of existence of one. With the intention of use in medical hospitals to aid communication between a deaf or hard-of-hearing patient with the doctor, the foremost goal was presenting a robust system, which does not impose any constraints on either the signer or the background. The proposed system combines skeletal data and depth information for hand tracking and segmentation, without relying on any color markers, or skin color detection algorithms. The extracted features describe the four elements of the hand that are used to describe the phonological structure of ArSL: articulation point, hand orientation, hand shape, and hand movement. Hidden Markov Model (HMM) was used for classification using ten-fold cross-validation, achieving an accuracy of 80.47%. Singer-independent experiments resulted in an average recognition accuracy of 64.61%.