Efficient rate-distortion optimization for HEVC using SSIM and motion homogeneity

Rate-distortion optimization is widely used in modern video codecs to make various encoder decisions in order to optimize the trade-off between bit-rate and quality. The distortion models used in HEVC are mean squared error (MSE) and sum of absolute difference (SAD), both of which are not always reflective of perceptual quality. In this paper, we show that SSIM, which has been found to be a good indicator of image visual quality, can be used as the distortion metric in the RDO framework in a simple yet effective manner by modifying the Lagrange multiplier used in RDO with the difference of motion homogeneity. Experiments show that the proposed scheme can achieve better rate-SSIM performance and provide higher subjective quality when compared with HEVC test model 10 (HM10.0) anchor.