Drift Characterization of Intra Prediction and Quantization in H.264

The combination of intra prediction, transform, and quantization in the H.264/AVC video coding standard offers high compression performance in picture areas where inter prediction is unavailable or fails. Motivated from coding experiments with the reference software that resulted in significant visual distortion with certain types of content, we investigated intra prediction further. Intra prediction may propagate distortion to subsequent blocks or macroblocks. As a result, rate-distortion optimized mode decision needs to account for this distortion. Furthermore, rounding operations in H.264/AVC accumulate small but non-trivial errors. This is especially true for the encoding process of the 16 times 16 intra prediction mode, which involves a combination of DCT and Hadamard transforms. We investigate, theoretically and experimentally, the problem of visual distortion caused by sub-optimal intra prediction coding decisions by modeling distortion propagation across subsequent blocks. We propose two algorithms that address the problem.