Rate Distortion Optimization of H.264 with Main Profile Compatibility

Using soft decision quantization rather than the conventional hard decision quantization, this paper studies a joint rate distortion design of motion prediction, quantization and entropy coding for the H.264 main profile encoding. Specifically, a soft decision quantization algorithm is proposed based on the context adaptive binary arithmetic coding method in H.264. The proposed algorithm is proved to achieve optimal soft decision quantization for a block with given motion prediction and quantization step size in the sense of minimizing the true rate distortion cost. It is then used in jointly designing motion prediction and residual coding for H.264 main profile coding. Experiments have been conducted based on the reference encoder JM82 of H.264. Comparative studies show that the proposed joint design method achieves an average 10% rate reduction while maintaining the same quality over the rate distortion method in the reference software of H.264