Mobile camera based cross-screen interaction by object matching and tracking

Heterogeneous devices with large displays have become popular in people lives. While conveying public information, most of them are lack of support for human interaction. This paper proposes a cross device interaction system for users to conveniently manipulate the content on public displays through personal mobile terminals with cameras. Users acquire, insert, and act on elements of the displayed content with simple operations on mobile terminals. Cross device elements targeting is done with an efficient visual based object matching and tracking algorithm when users capture the displayed contents as a live video. Augmented information for the elements is shown on the mobile terminal screen. The content in the large display can be operated by clicking, dragging, etc., on the mobile terminal. The proposed system can be applied to interactive advertisements, multi-user games and collaborative meetings. An evaluation consisting of various experiments demonstrated the usability and efficiency of the proposed system.