Depth map restoration and upsampling for kinect v2 based on IR-depth consistency and joint adaptive kernel regression

This paper presents a depth map restoration scheme for both the raw and projected depth map from Kinect v2 sensor. Based on IR-depth consistency, erroneous depth readings around foreground objects are removed by an edge aware consistency correction method. Moreover, a joint adaptive kernel regression algorithm is designed to upsample the sparse depth map after the projection from Kinect v2 sensor´s depth camera to its full HD video camera. The structural information in the high resolution color image is implicitly utilized to guide the upsampling of depth map. The effectiveness of the proposed upsampling algorithm is illustrated by experimental results and comparisons on both real Kinect v2 data and Middlebury dataset.