Linking attention to physical action in complex decision making problems

Embodied interaction concerns the way that user senses the environment, acquires information, and exhibits intention by means of physical action. In a complex decision making scenario, which requires maintaining high level of attention continuously and deep understanding about the task and its context, the use of embodied interaction has the potential to promote thinking and learning. Creating a framework that allows decision makers to interact with information using the whole body in intuitive ways may offer cognitive advantages and greater efficiency. This paper proposes such a computational framework based on a Bayesian approach (coined BAN) to infer operators´ focus of attention based on the operators´ physical expressions. Then, utility theory is adopted in order to determine the best combinations of interaction modalities and feedback. Experiments involving five physical interaction modalities (touchless, glove-based, and step gestures, speech, and body balance) and two feedback modalities (visual and sound) were conducted to assess the proposed framework´s performance. This also includes the likelihood of assessed attention from enhanced BANs and task performance as a function of the interaction and control modalities. Results show that physical expressions have a determining factor in the quality of the solutions in spatio-navigational type of problems.