A Kinect-based framework for better user experience in real-time audiovisual content manipulation

Applications for real-time multimedia content production, because of their delay-sensitive nature, require fast and precise control by the user. This is commonly achieved by specialized physical controllers that are application-specific with steep learning curves. In our work, we propose using the inputs from Microsoft Kinect as a controller interface. Originally introduced as a peripheral of XBox, Kinect is a multimodal device equipped with RGB Camera, Depth Sensor and Microphone Array. We use those inputs in order to provide a non-tactile controller abstraction to the user, targeting multimedia content creation. Current Kinect-based solutions, try to recognize natural gestures of the user, and classify them as controller actions. The novelty of our implementation is that instead of extracting gesture features, we directly map the inputs from the Kinect to a suitable set of values for the multimedia application. By removing the gesture recognition concept, we are able to create a generic and lightweight framework, with a clear interface to the user. We examine the usability of the framework through the development and evaluation, of a Kinect-controlled real-time multimedia application.