Free viewpoint video synthesis and presentation from multiple sporting videos

This paper introduces two kinds of free viewpoint observation systems for sporting events captured with uncalibrated multiple cameras in a stadium. In the first system (viewpoint on demand system), a user can watch the realistic sporting scenes with the original stadium. In the second system (mixed reality presentation system), a user can virtually watch the scenes overlaid on a desktop stadium model via a video see-through head mounted display (HMD). In the both systems, the user can observe sporting events from his/her favorite viewpoints, where the virtual view images are synthesized and presented by performing view interpolation. As outdoor scenes often have variations in lighting, we develop the systems to handle the changes of lighting condition. If the captured scene contains shadows, we synthesize the virtual view image of the shadows of the background and the foreground independently from real camera images using projective geometry between cameras. The shadows of the foreground objects are then overlaid on the synthesized background of the original stadium or the stadium model in front of the user respectively. The results indicate that the appearance of shadows can produce a realistic mixed reality presentation.