Real-time Sound Source Localization Based on Audiovisual Frequency Integration

We propose a pixelwise sound source localization algorithm based on audiovisual frequency integration. The localization is realized by detecting the common vibration dynamics of sound sources in the audio and the brightness signal. In order to detect the common vibration dynamics, temporal correlation values between the two signals are calculated in the algorithm. Several experimental results are shown for vibrated objects, and the pixelwise sound source localization images are obtained