Rate-distortion optimized transform for intra-frame coding

In this paper, a novel algorithm is proposed for intra-frame coding, named as rate-distortion optimized transform (RDOT). Unlike existing intra-frame coding schemes where the transform matrices are either fixed or mode dependent, in the proposed algorithm, transform is implemented with multiple candidate transform matrices. With this flexibility, for coding each residual block, the encoder is endowed with the power to select the optimal transform matrix in terms of rate-distortion tradeoff. The proposed algorithm has been implemented in the latest ITU-T VCEG-KTA software. Experimental results show that, over a wide range of test set, the proposed method achieves average 0.43dB coding gain compared with the recent Mode-Dependent Directional Transform (MDDT). The improvement is more significant at high bit-rates, and up to 1dB coding gain can be achieved.