An improved intra mode decision method for x264 video encoding based on hybrid distortion discriminated criteria

Distortion discriminated criteria play a very important role in rate-distortion optimization (RDO), which is an effective way of video encoding. In x264, the sum of the square differences (SSD) and the sum of the absolute differences (SAD) are used as quality metrics for RDO, while these two criteria are poorly correlated to human visual system. In this paper, the structural similarity index (SSIM) is used in the RDO scheme and an improved algorithm based on hybrid distortion discriminated criteria is proposed for x264 intra mode selection. With plenty of simulations, the RDO formula with SSIM is remodeled. The results show that the proposed method can provide better subject visual quality of the reconstructed video than the original x264.