Instant Mobile Video Search With Layered Audio-Video Indexing and Progressive Transmission

The proliferation of mobile devices is producing a new wave of applications that enable users to sense their surroundings with smart phones. People are preferring mobile devices to search and browse video content on the move. In this paper, we have developed an innovative mobile video search system through which users can discover videos by simply pointing their phones at a screen to capture a very few seconds of what they are watching. Different than most existing mobile video search applications, the proposed system is aiming at instant and progressive video search by leveraging the light-weight computing capacity of mobile devices. In particular, the system is able to index large-scale video data using a new layered audio-video indexing approach in the cloud, as well as generate lightweight joint audio-video signatures with progressive transmission and perform progressive search on mobile devices. Furthermore, we showcase that the system can be applied to two novel applications-video entity search and video clip localization. The evaluations on the real-world mobile video query dataset show that our system significantly improves user´s search experience due to search accuracy, low retrieval latency, and very short recording duration.