Low complexity RDO mode decision based on a fast coding-bits estimation model for H.264/AVC

Rate distortion optimization (RDO) is a very efficient tool for deciding a macroblock´s coding mode, which has been adopted by the H.264/AVC codec and brings higher coding efficiency. However, due to the precise calculation of the current macroblock´s distortion and coding-bits for each coding mode, RDO-based mode decision involves high computational complexity. In this paper, a simple model for quickly estimating the coding-bits of the quantized transform coefficients, which account for the major part of the computation when coding a macroblock, is presented. The model only uses nonzero coefficients´ magnitude and coordinates information by add operations to fulfill estimation so that computational complexity can be significantly reduced. Experimental results show that with this model the computational complexity of RDO-based mode decision in the H.264/AVC codec can be reduced by 40-60% while coding efficiency has little drop.