Fast prediction mode decision with hadamard transform based rate-distortion cost estimation for HEVC intra coding

The emerging video coding standard, High Efficiency Video Coding (HEVC), aims at doubling coding efficiency of H.264/AVC. In the intra encoding, Rate-Distortion Optimization (RDO) processes are employed to determine the best prediction mode. RDO based mode decision accounts for 35-39% coding time, because the large scale two-dimensional(2D) DCT/IDCT introduces a plethora of computation- and hardware-consuming multiplications. In this paper, the simplified Rate-Distortion (RD) cost estimation algorithms, which are based on the Hadamard transform and the distortion evaluation without the signal reconstruction, are proposed. When embedded to HEVC test Model(HM-5.2), our methods averagely achieve 16.1% time saving at the price of 0.055dB BD-PSNR loss, or equivalently 1.27% BD-BR increasing in intra coding. The corresponding VLSI design of the proposed algorithms is implemented with TSMC 90nm 1P9M technology. The maximum clock speed is 418 MHz under the worst work conditions (125°C, 0.9V). As compared with the primitive design, 64.9% hardware cost can be saved by our schemes. One proposed engine fulfills the throughput for 4K-UHDTV@28fps real-time encoding.