Probabilistic 3-D motion estimation for rolling shutter video rectification from visual and inertial measurements

Video acquired by handheld CMOS cameras may suffer from rolling shutter artifacts. Rolling shutter artifacts, which are due to the rows in the image sensor array being exposed sequentially from top to bottom, increase with the speed of the relative motion between the scene and camera. To rectify these artifacts, one needs to recover the projection parameters for each row. In this paper, we propose a probabilistic method to estimate 3-D camera rotation by using video and inertial measurements on the handheld platform, such as a smart phone. Our contributions are (1) an efficient sensor fusion algorithm using an extended Kalman filter, and (2) a quality assessment method using vanishing point detection. Experiments indicate that the proposed sensor fusion algorithm produces a more accurate orientation estimate and better rectifies rolling shutter artifacts.