Multi2Rank: Multimedia Multiview Ranking

Multimedia retrieval is a search and ranking task defined over multiple modalities. These modalities include speech, image, and text, which provide different views of the multimedia object. Queries to a multimedia retrieval system often take the form of a text only query and return a ranked result set which combines these multiple views. The text only query includes multiple phrases which identify features of a specific view. This multiview problem presents a challenge in mapping these phrases into the correct view feature space. A second challenge for the multimedia retrieval system is in building a ranking model which considers the unique feature space of each view. In this paper, we propose a hierarchical multimedia multiview rank learning model, called Multi²Rank, to overcome the challenges of this unique ranking problem. The first layer of our model uses natural language processing techniques to identify view specific phrases and output a ranked mapping of the phrases into their respective views. Next, we model the individual feature space for each multimedia view and create a view specific model using gradient boosted regression trees. The ranked set from each unique view is then passed to the final layer of the hierarchy, where the model generates a final ranked result set. We show that our multiview rank learning approach is effective by evaluating the methods using a large Internet video repository, queries, and ground truth, from the TRECVid evaluations.