Fusing Kinect sensor and inertial sensors with multi-rate Kalman filter

This paper presents a sensor fusion approach to fusing Microsoft Kinect sensor and the built-in inertial sensors in a mobile device. A multi-rate Kalman filter is designed and applied for fusing the low-sampling-rate (30Hz) uncertain positions sensed by the Kinect sensor and the high-sampling-rate (90Hz) accelerations measured by the inertial sensors. These sensors have complementary properties. The Kinect can be applied for skeleton tracking, which gives the joints´ positions. Meanwhile, the built-in inertial sensors in the mobile device sense the hand motion and the acceleration can be estimated through inertial sensor fusion. Firstly, convert the acceleration estimated with inertial sensors from the body frame into the Kinect coordinate system. Experimental results show that the hand accelerations estimated with the Kinect sensor and the inertial sensors are comparable. Secondly, design and apply a multi-rate Kalman filter for sensor fusion. The sensor fusion helps improve the accuracy of the system state estimation including the position, the velocity and the acceleration. This is of great benefit for combining inertial sensors and the external position sensing device for indoor augmented reality (AR) and other location-aware sensing applications.