A perceptual rate-distortion optimization approach based on piecewise linear approximation for video coding

The core question in Perceptual Rate Distortion Optimization is how to find the Lagrange multiplier λ, which essentially requires the construction of a perceptual quality based RD model. A compelling RD modeling expects that the model reflects the best achievable Rate Distortion trade-off and captures the RD behavior with high accuracy. To this end, we rescale the λ associated with MSE-RDO to match the dynamic range of the perceptual distortion and populate a family of RD samples from Perceptual-RDO with its Lagrange multiplier as offsetting the rescaled λ. Considering that the envelope curve that encloses all the RD samples depicts the best achievable RD bound, we propose a piecewise linear approximation approach (since there is no guarantee that the RD model can be accurately fitted by a single function e.g. D = ae^bR, D = aR^b) to represent it. Each segment in the piecewise line is obtained with fair computational complexity: fitting the RD samples given the same QP with circle curve and finding the common tangent line over two adjacent circle curves. Experiments show that our proposed approach can reduce bitrate by 2-5% given the same perceptual distortion than the conventional approach.