Adaptive quantization with balanced distortion distribution and its application to H.264 intra coding

Quantization in H.264 is achieved in the DCT domain using scalar quantizers, which assume a sum distortion constraint and often produce considerably larger distortions on block boundaries than inside a block in the pixel domain. This biased distortion distribution degrades the rate distortion (RD) performance of H.264 intra coding whose prediction is exclusively based on boundary pixels. This paper considers the problem of designing balanced distortion quantizers (BDQs) in the DCT domain, which, in addition to the sum distortion constraint, require evenly distributed distortions in the pixel domain. In a special case where DCT coefficients are independent Gaussian, the problem is solved as a convex optimization problem. Using this approach, we design BDQs and apply them to improve H.264 intra coding. Experimental results on typical frames show that the improved intra coding scheme consistently outperforms its counterpart in H.264 main-profile, averaging 5-9% rate reduction for QCIF frames, and 7-12% for CIF frames with aligned distortions.