Constrained 3D Rotation Smoothing via Global Manifold Regression for Video Stabilization

We present a novel motion smoothing algorithm for hand-held cameras with application to video stabilization. Video stabilization seeks to remove unwanted frame-to-frame jitter due to camera shake. For video stabilization, we use a pure 3D rotation motion model with known camera projection parameters. The 3D camera rotation can be reliably tracked by a gyroscope as commonly found on a smart phone or tablet. In this paper, we directly smooth the sequence of camera rotation matrices for the video frames by exploiting the Riemannian geometry on a manifold. Our contributions are 1) formulation of motion smoothing as a geodesic-convex constrained regression problem on a nonlinear manifold based on geodesic distance, 2) computation of gradient and Hessian of the objective function using Riemannian geometry for gradient-related manifold optimization, and 3) generalization of the two-metric projection algorithm in Euclidean space to manifolds to solve the proposed manifold optimization problem efficiently. The geodesic-distance-based smoothness metric better exploits the manifold structure of sequences of rotation matrices. The geodesic-convex constraints effectively guarantee that no black borders intrude into the stabilized frames. The proposed manifold optimization algorithm can find the global optimal solution in only a few iterations. Experimental results show that video stabilization based on our motion smoothing algorithm outperforms state-of-the-art methods by generating videos with less jitter and without black borders.