Mosaicing with Parallax using Time Warping

2D image alignment methods are applied successfully for mosaicing aerial images, but fail when the camera moves in a 3D scene. Such methods can not handle 3D parallax, resulting in distorted mosaicing. General egomotion methods are slow, and do not have the robustness of 2D alignment methods. We propose to use the x-y-t space-time volume as a tool for depth invariant mosaicing. When the camera moves on a straight line in the x direction, a y-t slice of the space-time volume is a panoramic mosaic, while a x-t slice is an EPI plane. Time warping, which is a resampling of the t axix, is used to form straight feature lines in the EPI planes. This process will simultaneously give best panoramic mosaic in the y-t slices. This approach is as robust as 2D alignment methods, while giving depth invariant motion ("ego motion"). Extensions for two dimensional camera motion on a plane are also described, with applications for 2D mosaicing, and for image based rendering such as "light field".