3D Sound Localization Based on Audio and Video

In this paper, 3D sound localization is presented by fusing audio information with video signals. This localization system consists of two subsystems, an audio setup with 3 microphones in a triangular format and a video source (a camera), all mounted on a pan-tilt unit. The audio subsystem is to generate two independent time differences ΔT1and ΔT2. Based on the set of independent time differences, a 3D curve on which the sound source lies can be derived based on the setup alone. On the other hand, the video subsystem gives a plane on which the sound source lies. The intersection between the 3D curve and the plane can be used to determine exact location, i.e., a unique solution, of the sound source. As it isfairly complicated for such an analytical approach, neural network fusion is then proposed to fuse the two systems together conveniently and seamlessly. Simulation results show that this sound localization system can locate the sound source with satisfactory accuracy.