Multimodal Temporal Panorama for Moving Vehicle Detection and Reconstruction

In this work, we present a multimodal temporal panorama (MTP) representation that synchronizes visual, motion, and acoustic signatures of moving vehicles in the time axis. The MTP representation includes two layers: a synopsis layer and a snapshot layer. The temporal synopsis consists of 1) a panoramic view image (PVI) to represent vehicles´ presence, which is constructed from 1D vertical detecting lines of a selected column location of all video frames, 2) an epipolar plane image (EPI) to characterize their motion (speeds and directions), generated from 1D horizontal scanning lines along the vehicles´ moving paths, and 3) an audio wave scroll for visualizing moving vehicles´ acoustic signatures. The MTP synopsis not only synchronizes all the three modalities (visual, motion and acoustic) of the vehicles, but also provides information that can perform automatic detection tasks including moving vehicle visual detection, motion estimation, and acoustic signature retrieval. Then in the snapshot layer, the occlusion-free, motion-blur-free, and view-invariant reconstruction of each vehicle (with both shape and motion information) and its acoustic signatures (e.g. spectrogram) are embedded. The MTP provides a very effective approach to (semi-)automatically labeling the multimodal data of uncontrolled traffic scenes in real time for further vehicle classification, check-point inspection and traffic analysis. The concept of MTP may not be only limited to visual, motion and audio modalities, it could also be applicable to other sensing modalities that can obtain data in the temporal domain.