Multi-modal encoding of speech in memory: a first report

Why do people like to watch videos on TV? Why is there now increased interest in video telephones and multimedia technologies that were developed back in the 1960´s? Obviously, the availability of new digital technology has played an enormous role in this transition. But, we also believe this is in part due to the same operating principle that encourages listeners in noisy environments to orient toward a talker´s face. A multimodal speech signal is extremely robust and informative and provides information that perceivers are able to exploit during perceptual analysis. We present results from two experiments that examined performance in immediate memory and serial recall tasks with normal hearing listeners using unimodal (auditory only) and multimodal (auditory+visual) presentation. Our findings suggest that the addition of visual information in the stimulus display about the speakers´ articulation affects the efficiency of initial encoding operations at the time of perception and also results in more detailed and robust representations of the stimulus events in memory. These results have implications for current theories of speech perception and spoken language processing