A real-time interactive nonverbal communication system through semantic feature extraction as an interlingua

There has been a growing interest in the use of networked virtual environment (NVE) technology to implement telepresence that allows participants to interact with each other in shared cyberspace. In addition, nonverbal language has attracted increased attention because of its association with more natural human communication, and especially sign languages play an important role for the hearing impaired. This paper proposes a novel real-time nonverbal communication system by introducing an artificial intelligence method into the NVE. We extract semantic information as an interlingua from the input text through natural language processing, and then transmit this semantic feature extraction (SFE) to the three-dimensional (3-D) articulated humanoid models prepared for each client in remote locations. Once the SFE is received, the virtual human is animated by the synthesized SFE. Experiments with Japanese and Chinese sign languages show this system makes the real-time animation of avatars available for the participants when chatting with each other. The communication is more natural since it is not just based on text or predefined gesture icons. This proposed system is suitable for sign language distance training as well.