Automatic sign segmentation from continuous signing via multiple sequence alignment

In order to build a sign language recognition framework, one needs to collect sign databases that contain multiple samples of isolated signs, which is a hard and time consuming task. In this study, our aim is to obtain such a database by automatically extracting isolated signs from continuous signing, recorded from the broadcast news for the hearing-impaired. We present an unsupervised, multiple alignment-based approach for sign segmentation. Among the modalities used to form a sign, hand gestures carry most of the information, manifested as hand motion and shape. To handle these two sources of information, we experimented with different feature sets, with different fusion methods on different alignment approaches: feature concatenation on Dynamic Time Warping (DTW) and Hidden Markov Models (HMMs), modeling via coupled and parallel HMMs, and sequential fusion of DTW and HMM. Our experiments on Turkish broadcast news videos show that (1) using low level shape descriptors is suitable for the alignment task, (2) the highest accuracy is obtained by modeling the signs with HMM using the intervals found previously by DTW.