Rate distortion optimized quantization based on weighted mean squared error for lossy image coding

This paper considers the problem of quantization in image coding where quality loss is measured by weighted mean squared error (WMSE) in an attempt to maintain structural similarity (SSIM). A rate distortion optimized quantization (RDOQ) scheme based on the WMSE is described and integrated in libjpeg. Experimental results on standard test images show that libjpeg with the RDOQ scheme achieves superior rate-SSIM performance against libjpeg (with or without MSE-based RDOQ) and the more recent WebP, while maintaining full JPEG compatibility.