Dante vision: In-air and touch gesture sensing for natural surface interaction with combined depth and thermal cameras

Researchers have paid considerable attention to natural user interfaces, especially sensing gestures and touches upon an un-instrumented surface from an overhead camera. We present a system that combines depth sensing from a Microsoft Kinect and temperature sensing from a thermal imaging camera to infer a variety of gestures and touches for controlling a natural user interface. The system, coined Dante, is capable of (1) inferring multiple touch points from multiple users (92.6% accuracy), (2) detecting and classifying each user using their depth and thermal footprint (87.7% accuracy), and (3) detecting touches on objects placed upon the table top (91.7% accuracy). The system can also classify the pressure of chording motions. The system is real time, with an average processing delay of 40 ms.