A study on the rate distortion modeling for High Efficiency Video Coding

This paper gives a study on the rate distortion modeling for High Efficiency Video Coding (HEVC) and new rate distortion models are proposed by considering the coding framework change of HEVC. Although HEVC is also characterized by the traditional prediction/transform hybrid coding framework with more improved coding tools, e.g. large size coding unit (CU), prediction unit (PU) and transform unit (TU), two new coding modules are added as sample adaptive offset(SAO) and adaptive loop filter(ALF). SAO and ALF are added after the motion compensation reconstruction, which would reduce the distortion of the reconstructed image. With considerations on these changes, new rate distortion models are proposed. For rate modeling, this paper proposes a separate model for header bits and coefficient bits respectively. For distortion modeling, the quantization distortion and the distortion reduction induced by SAO/ALF are modeled jointly. The experimental results show that the coding rate and distortion can be estimated by the proposed model accurately. These models can be used for rate distortion cost estimation for mode decision or rate control in the future.