Rate-Complexity-Distortion Optimization for Hybrid Video Coding

In recent years, video applications on handheld devices became more and more popular. Due to limited computational capability and power supply in handheld devices, rate-complexity-distortion optimization (RCDO) algorithms at encoder side draw increasing attention. The target of RCDO is to obtain the best rate-distortion (R-D) performance under a constraint of complexity. Generally, there are three essential problems in RCDO. First, complexity needs to be properly mapped to a target in terms of coding parameters such that the control over complexity can be achieved. Second, the complexity budget should be efficiently distributed among frames or other coding units. Third, the allocated budget for each coding unit has to be effectively used to obtain good R-D performance. In this paper, these problems are well addressed. To obtain a large dynamic range in complexity control, medium-granularity control methods are presented. Then, a frame level complexity allocation algorithm is developed based on dependent rate-distortion function. Finally, an adaptive mode and reference searching method is proposed for motion compensation process. Comprehensive simulations verify the proposed algorithms. In the environment of the H.264/AVC reference software, an average gain of over 0.5 dB and 0.7 dB in BD-PSNR was achieved for nine sequences at low complexity when compared to two RCDO methods from literature. Moreover, experiments on x264 (a practical implementation of H.264/AVC) show that the proposed algorithms outperform predefined complexity levels by x264 in terms of both coding efficiency and computational scalability.