Modality and contextual differences in computer based non-verbal communication training

The number of people who have trouble with social skills and communication is now greater than ever for a variety of reasons. An extreme case of these traits is characteristics of autism spectrum disorders. Our objective is to enable people with social and communication difficulties to improve their social and communication skills. Previous reports have shown modality and contextual information is useful for emotion recognition. This paper examines modality and contextual differences in the context of the existing social skills training aid NOCOA+. In NOCOA+, there are four types of modalities: audiovisual, visual, audio, and verbal. It also has three types of contextual information: no context, prior 5 seconds context, and prior 10 seconds context. We performed two experiments that examined the effect of modality and context on social skills training tools. The results elucidated characteristics of non-verbal behaviour incorporating each sensory modality, and also found that contextual information is helpful to infer non-verbal behaviour.