Model-aided coding: using 3-D scene models in motion-compensated video coding

We show that traditional waveform coding and 3-D model-based coding are not competing alternatives but should be combined to support and complement each other. Both approaches are combined such that the generality of waveform coding and the efficiency of 3-D model-based coding are available where needed. The combination is achieved by providing the block-based video coder with a second reference frame for prediction which is synthesized by the model-based coder. Since the coding gain of this approach is directly related to the quality of the synthetic frame, we have extended the model-aided coder to cope with illumination changes and multiple objects. Remaining model failures and objects that are not known at the decoder are handled by standard block-based motion-compensated prediction. Experimental results show that bit-rate savings of up to 45% are achieved at equal average PSNR when comparing the model-aided codec to TMN-10, the test model of the H.263 standard