Sound influences visual attention discriminately in videos

In the long term research of visual attention, various computational models have been proposed. However, most of those works do not take audio into consideration. In practice, visual signals often come along with audio. Therefore, it is natural to investigate the influence of audio on visual attention. In this paper, we focus on the problem of when will audio influence visual attention during video watching. We perform eye-tracking experiments on a set of 60 videos in audio-visual (AV) and visual (V) conditions. Based on the eye movement data, we find that the influence of audio on visual attention depends on the consistency between the visual and audio signals. If the salient objects from the visual perspective are not consistent with the salient objects from the audio perspective, audio will influence visual attention. Otherwise, audio has little influence on visual attention.