On fast coding tree block and mode decision for high-Efficiency Video Coding (HEVC)

In the current HEVC test model (HM), a quad-tree based coding tree block (CTB) representation is used to signal mode, partition, prediction and residual information. The large number of combinations of quad-tree partitions and modes to be tested during rate-distortion optimization (RDO) results in a high encoding complexity. In this paper, we investigate and compare a variety of algorithms for fast CTB and mode decision. Experimental results from HM4-based implementations show that different strategies can provide a range of complexity-performance trade-offs. In particular, our proposed CU Depth Pruning algorithm can reduce encoding time by about 10% with only 0.1% coding loss, while a combination of our proposed Early Partition Decision and an early CU termination approach can reduce encoding time by about 40% with about 1% coding loss.