Online learning and integration of complex action and word lexicons for language grounding

This paper presents a computational framework for the development and integration of action and language capabilities through symbol grounding. Our approach is based around the fundamental technique of building lexicons of perceptual “simulators” which link noisy sensory experiences to internal symbolic representations. These sensory representations are paired with a basic model of association, allowing for the grounding of linguistic symbols directly in action knowledge - a grounding which is then exploited to bootstrap the development of more advanced capabilities. The performance of this computational framework is tested in the context of online tutoring scenarios using the iCub robotic platform. Such an experimental environment requires development of techniques and algorithms suited for incremental learning and real-time processing.