A multisensory identification system for robotics

We present a new multimodal object identification system for a robot. Humans constantly face complex multimodal identification tasks with success. This work is based on a cognitive theory of multisensory integration, suggesting convergence of specific unimodal sensory pathways into heteromodal areas, and feedback influences from multimodal to unimodal processing. To implement the multisensory identification system of the robot, two suitable connectionist networks have been linked into a two-level modular architecture. The first level is made of small and fast incremental neural classifiers that recognize independently modality-specific inputs. The second level is a recurrent neural network: a multiple bidirectional associative memory that integrates outputs of each first level sub-system. These two levels cooperate in both forward and backward ways to identify the object perceived. This model has been tested with a virtual robot navigating in a multi-modal environment, where objects are composed of images with sounds. The inputs of the robot are dynamically generated according to the position of the robot, its orientation, and its perceptive fields