Automatic Testing of Natural User Interfaces

Automated test generation can effectively explore programs through their programmer interfaces and traditional graphical user interfaces, but the recent advent of natural user interfaces (NUI) based on motion and gesture detection, for example the Microsoft Kinect, has outrun software testing research. This leaves a rapidly growing domain of software ranging from entertainment to medical applications without suitable test automation techniques. To address this issue, we propose a technique that automatically tests Kinect-based applications by synthesising realistic sequences of skeletal movement. The novel test cases are generated by a statistical model, which is trained on a corpus of common gestures. Evaluation on a gesture-controlled Kinect web browser application demonstrates that our approach achieves significantly higher code coverage than random test inputs.