Efficient intra-4×4 mode decision based on bit-rate estimation in H.264/AVC

Rate-distortion optimization (RDO) technique is widely employed by H.264/AVC for the purpose of determining the best mode. However, such technique results in dramatic increase in the computation complexity of the underlying encoder. In this paper, we address this problem by presenting an efficient intra-4x4 mode decision algorithm. The algorithm works by approximating the bit-rate so as to reduce the computational cost of RDO and the main idea is the following: First we have found a quick way to estimate the bit-rate via the number of DCT coefficients to be quantized to 0 and that to be quantized to plusmn1. The parameters of the estimated function are adaptively obtained by using the Least Squares Fitting method of the above and the left block in the current frame and the co-location one in the previous encoded frame as the feedback. This close loop bit-rate estimation would skip the processes of quantization, inverse transform, entropy coding and reconstruction; we then use the estimated bit-rate and Sum of Absolute Differences (SAD) to simplify the optimization process of R-D cost function. Experimental results show that our scheme decreases the time for intra coding by 50% with negligible loss of PSNR, and that comparing with those fast mode-decision algorithms based-on local edge direction, the optimal prediction mode obtained via our scheme is closer to that obtained via the original RDO in statistic.