A novel rate-distortion optimization method of H.264/AVC intra coder

In H.264/AVC, the concept of rate-distortion optimization (RDO) mode decision has proven to be an important coding tool. But the complexity and computation load of RDO technique is extremely high. In this paper, we propose an enhanced low complexity cost function for H.264/AVC intra 4x4 mode selections. The enhanced cost function uses sum of absolute Hadamard-transformed differences (SATD) and variance of the residual block to estimate distortion part of the cost function. A threshold based large coefficients count is also used for estimating the bit-rate part. The proposed method improves the rate-distortion (RD) performance of the conventional fast cost functions while maintaining low complexity requirement.