User attention oriented augmented reality on documents with document dependent dynamic overlay

When we read a document (any kind of, scientific papers, novels, etc.), we often encounter a situation that the information from the reading document is too less to comprehend what the author(s) would like to convey. In this paper, we demonstrate how the combination of a wearable eye tracker, a see-through head-mounted display (HMD) and an image based document retrieval engine enhances people´s reading experiences. By using our proposed system, the reader can get supportive information in the see-through HMD when he wants. A wearable eye tracker and a document retrieval engine are used to detect which line in the document the reader is reading. We propose a method to detect the reader´s attention on a word in a reading document, in order to present information at a preferable moment. Furthermore, we also propose a method to project a point of the document to a point of the HMD screen, by calculating the pose of the reading document in the camera image. This projection enables the system to overlay the information dynamically in an augmented view on the reading line. The results from the user study and the experiments show the potential of the proposed system in a practical use case.