Rate-Distortion Optimized Transforms Based on the Lloyd-Type Algorithm for Intra Block Coding

The directional intra prediction (IP) in H.264/AVC and HEVC tends to cause the residue to be anisotropic. To transform the IP residue, Mode Dependent Directional Transform (MDDT) based on Karhunen Loève transform (KLT) can achieve better energy compaction than DCT, with one transform assigned to each prediction mode. However, due to the data variation, different residue blocks with the same IP mode may not have the same statistical properties. Instead of constraining one transform for each IP mode, in this paper, we propose a novel rate-distortion optimized transform (RDOT) scheme which allows a set of specially trained transforms to be available to all modes, and each block can choose its preferred transform to minimize the rate-distortion (RD) cost. We define a cost function which is an estimate of the true RD cost and use the Lloyd-type algorithm (a sequence of transform optimization and data reclassification alternately) to find the optimal set of transforms. The proposed RDOT scheme is implemented in HM9.0 software of HEVC. Experimental results suggest that RDOT effectively achieves 1.6% BD-Rate reduction under the Intra Main condition and 1.6% BD-Rate reduction under the Intra High Efficiency (HE) 10bit condition.