Interactive user oriented visual attention based video summarization and exploration framework

An interactive user oriented high definition visual attention based video summarization and exploration framework is proposed to extract feature frames from a video collection and allow users to interactively explore those feature frames. It is based on previous work [1] that applies high definition visual attention algorithm mapping and multivariate mutual information to select a feature frames to represent each shot, then uses a self-organizing map to remove the redundant frames. After the video summary process, the extracted feature frames are connected into a network structure. Each node contains the information of the feature frame and the relation to other nodes. The relation between nodes are defined by clustering algorithms (self-organizing map, k-means, support vector machine, etc), expert systems (look-up table, fuzzy logic statement, etc) or any algorithm that defines similarity (sift, surf, etc). When a user select one node, depending on the user setting, the related nodes will be displayed onto a 2D canvas. In this way user is be able to interactively to browse through the whole video collection.