DocumentCode :
3748936
Title :
Weakly-Supervised Alignment of Video with Text
Author :
P. Bojanowski;R. Lajugie;E. Grave;F. Bach;I. Laptev;J. Ponce;C. Schmid
fYear :
2015
Firstpage :
4462
Lastpage :
4470
Abstract :
Suppose that we are given a set of videos, along with natural language descriptions in the form of multiple sentences (e.g., manual annotations, movie scripts, sport summaries etc.), and that these sentences appear in the same temporal order as their visual counterparts. We propose in this paper a method for aligning the two modalities, i.e., automatically providing a time (frame) stamp for every sentence. Given vectorial features for both video and text, this can be cast as a temporal assignment problem, with an implicit linear mapping between the two feature modalities. We formulate this problem as an integer quadratic program, and solve its continuous convex relaxation using an efficient conditional gradient algorithm. Several rounding procedures are proposed to construct the final integer solution. After demonstrating significant improvements over the state of the art on the related task of aligning video with symbolic labels [7], we evaluate our method on a challenging dataset of videos with associated textual descriptions [37], and explore bag-of-words and continuous representations for text.
Keywords :
"Data models","Hidden Markov models","Natural languages","Manuals","Speech","Optimization methods","Minimization"
Publisher :
ieee
Conference_Titel :
Computer Vision (ICCV), 2015 IEEE International Conference on
Electronic_ISBN :
2380-7504
Type :
conf
DOI :
10.1109/ICCV.2015.507
Filename :
7410864
Link To Document :
بازگشت