Fast mode decision for adaptive prediction error coding

Adaptive prediction error coding (APEC) in spatial and frequency domain is proposed and significantly improves the coding efficiency of video coders. However, this approach comes with the expense of increased encoding complexity because of the need to perform rate distortion optimization (RDO) for each block, to decide whether the block is coded in spatial or frequency domain. In this work, we propose novel fast mode decision algorithm operating on macroblock and block level to reduce the encoding complexity of APEC. The proposed algorithm estimates the edge orientation of the residual block using spatial domain filtering, and based on the estimated edge orientation, reduces the number of candidates to be checked in RDO. Furthermore, proposed method includes an early termination step that stops searching the best candidate based on the difference between spatial and frequency domain coding. Experimental results show that the proposed fast mode decision algorithm can achieve averagely 90.7% encoding time reduction of APEC with only about 0.06 dB loss in coding efficiency.