Efficient rate-distortion optimized mode selection of H.264/AVC intra coding

Rate-distortion optimization (RDO) can significantly improves encoder performance in H.264-like video coding applications. In H.264/AVC, rate-distortion optimization (RDO) method has to code the video by exhaustively trying all the mode combinations including the different intra- and inter-prediction modes. Therefore, the complexity and computation load of video coding in H.264/AVC increase drastically compared to any previous standards. This paper reviewed the conventional fast cost functions of the intra encoder and proposed an enhanced low complex cost function for H.264/AVC intra 4×4 mode selections. The enhanced cost function uses sum of absolute Hadamard-transformed differences (SATD) and mean absolute deviation of the residual block to estimate distortion part of the cost function. A threshold based large coefficients count is also used for estimating the bit-rate part. The proposed method improves the rate-distortion performance of the conventional fast cost functions while maintaining low complexity requirement. As the results, the encoding process can be significantly accelerated with use of the proposed cost function. Simulation results confirmed that the proposed method reduced about 85% of computation of original encoder with negligible rate-distortion performance degradation.