Towards Improving Social Communication Skills With Multimodal Sensory Information

How to improve social communication skills for children, especially those with social communication difficulties such as attention deficit/hyperactivity disorder, has long been a challenge faced by researchers and therapists. Recent research indicates that computer-assisted approaches may be effective in addressing this issue. This study aimed to understand children´s behaviors and then provide appropriate support to improve their social communication skills. We have established an intelligent system, inside which a child can freely play interactive social skills games with virtual characters. The virtual characters can adjust their own behaviors by adapting to the child´s cognitive state (e.g., focus of attention) and affective state (e.g., happiness or surprise). The child´s behavior is identified in real-time by recognition of multimodal sensory information, which includes head pose and eye gaze estimation, gesture detection, and affective state detection supported by a series of algorithms proposed in this study. Furthermore, this intelligent system has been enabled in a nonintrusive manner using a novel approach of multicamera surveillance to provide the child with natural interaction with the system. Experimental results show the system can estimate a user´s attention and affective states with correctness rates of 93% and 91.3%, respectively. The results obtained suggest that the methods have strong potential as alternative methods for sensing human behavior and providing appropriate support.