Syncing Shared Multimedia through Audiovisual Bimodal Segmentation

This work emanates from the particularities residing in contemporary social media storytelling, where multiple users and publishing channels capture and share public events, experiences, and places. Multichannel presentation and visualization mechanisms are pursued along with novel audiovisual mixing (such as time-delay-compensation enhancement, perceptual mixing, quality-based content selection, linking to context-aware metadata, and propagating multimedia semantics), thus promoting multimodal social media editing, processing, and authoring. While the exploitation of multiple time-based media (audio and video) describing the same event may lead to significant content enhancement, difficulties regarding detection and temporal synchronization of multimedia events have to be overcome. In many cases, one can identify events based only on audio features, thus performing an initial cost-effective annotation of the multimedia content. This article introduces a new audio-driven approach for temporal alignment and management of shared audiovisual streams. The article presents the theoretical framework and demonstrates the methodology in real-world scenarios. This article is part of a special issue on social multimedia and storytelling.