Residue role assignment based transform partition predetermination on HEVC

The transform unit (TU), combining with the rate-distortion optimized quantization (RDOQ), as a part of the important features of HEVC brings much performance gain by adding more computation burden to the encoder. Different from the previous standards, the transform and quantization (T&Q) lead into a quadtree structure which requires to be split and calculated recursively for one coding unit (CU). The complexity issues limit the development of both software and hardware video code engine. To solve this problem, this paper proposes a residue role assignment based transform partition predetermination method by introducing the all zero block detection principle. The proposal generates from the observation that the rate-distortion optimized TU partitions generally depend on the distribution of the low frequency residue blocks. 3 different roles are assigned to the residue blocks based on 3 level of residue frequency value. Each role of residue relates to a concrete T&Q calculation. Therefore, instead of traversal calculating of full T&Q size and T&Q depth, the residue blocks are categorized previously and only the corresponding T&Q are calculated. The evaluation results state that the proposal can achieve 12.8% and 26.3% total encoding time reduction with only 1.00% and 0.80% BD bit-rate loss in average with both RDOQ off and on cases.