An interactive Taiwan sign language learning system based on depth and color images

Sign language is crucial for communicating with hearing-impaired people. This study intended to develop a new interactive Taiwan sign language learning system. The Microsoft Kinect was used as an input device, with which the color image, depth image, and skeleton points of hands and body were collected. We developed algorithms for extraction and classification of hand location, hand shape, and hand motion trajectory, and the above information was integrated to decipher the sign language. Experimental results confirmed the high recognition rate of the proposed method. We also developed a user-friendly interface with various modes to facilitate the learning of Taiwan sign language.