A hierarchical variable-sized block transform coding scheme for coding efficiency improvement on H.264/AVC

In this paper, a rate-distortion optimized variable block transform coding scheme is proposed based on a hierarchical structured transform for macroblock (MB) coding with a set of the order-4 and -8 integer cosine transform (ICT) kernels of H.264/AVC as well as a new order-16 ICT kernel. The set of order-4, -8 and -16 ICT kernels are applied for inter-predictive coding in square (4 × 4, 8 × 8 or 16 × 16) or non-square (16 × 8 or 8 × 16) transform for each MB in a hierarchical structured manner. The proposed hierarchical variable-sized block transform scheme using the order-16 ICT kernel achieves significant bitrate reduction up to 15%, compared to the High profile of H.264/AVC. Even if the number of candidates for the transform types increases, the encoding time can be reduced to average 4-6% over the H.264/AVC.