An Effective Method to Enhance the RDO Performance for H.264/AVC Video Coding

Rate-distortion optimization (RDO) plays quite an important role in today´s video coding scheme. This approach seeks a balance between the coding rate and distortion of the picture. The rate of video sequences is negatively related to the distortion between the original pictures and the reconstructed frames in traditionally opinions, for both of them are greatly affected by the quantization parameter (QP) - the larger QP means the greater quantization error, which leads to less bit-rate and more distortion. However, the deduced conclusion maintains the feature that the more precise prediction may benefit both the performance of the rate and that of distortion, with other conditions unchanged. The proposed scheme derives a theoretical estimation an effective algorithm to generate the reference data with least mean square error (MSE) is raised afterwards , which can effectively reduce the prediction error both in IPPPP sequence mode and in IBBBB style. Experimental results provide evidence for substantial improvements (about 0.3-0.5 dB gain) than JM18.3 (the most recent H.264/AVC reference software).