Fast Mode Decision for H.264/AVG using Mode and RD Cost Prediction

In an H.264/AVC encoder, each macroblock can be coded in one of a large number of coding modes, which requires a huge computational effort. In this paper, we present a new method to speed up the mode decision process using RD cost prediction in addition to mode prediction. In general, video coding exploits spatial and temporal redundancies between video blocks, in particular temporal redundancy is a crucial key to compress a video sequence with little loss of image quality. The proposed method determines the best coding mode of a given macroblock by predicting the mode and its rate-distortion (RD) cost from neighboring MBs in time and space. Compared to the H.264/AVC reference software, the simulation results show that the proposed method can save about 60% of the number of RD cost computations resulting in up to 57% total encoding time reduction with up to 3.5% bit rate increase at the same PSNR.