Tracking based motion segmentation under relaxed statistical assumptions

We present a novel and efficient motion segmentation and tracking algorithm that follows the shift and align paradigm. We introduce two statistical tests to evaluate the similarity of aligned image pixels or patches and we use them to determine the spatial extend of each segment. The one statistical test is fast and accurate when the noise is moderate and the other employs a sophisticated noise model involving the Mahalanobis distance to handle correlated noise. Direct computation of the Mahalanobis distance is prohibitively expensive so we apply the Sherman–Morrison–Woodbury identity and amortization to reduce the cost by several orders of magnitude. We tested both versions of the algorithm on a variety of image sequences (indoor and outdoor, real and synthetic, constant and varying lighting, stationary and moving camera, one of them with known ground truth) with very good results.