Efficient CABAC Rate Estimation for H.264/AVC Mode Decision

In H.264/AVC coders, the optimal mode decision for each coding unit is achieved based on a calculation of the rate-distortion (R-D) cost for each candidate mode. In this letter, we propose a context-adaptive binary arithmetic coding (CABAC)-based rate estimation scheme that reduces the computational complexity of the R-D cost function evaluation. The complex and time-consuming calculation of the arithmetic coding is replaced by a table lookup based on the proposed CABAC rate model for fast rate estimation. The context modeling part of CABAC is also simplified to accelerate the rate estimation process. The experimental results show that the proposed estimator follows the actual rate with the coefficient of determination R ² = 0.9997. The proposed estimator yields about a 30% reduction in the computational complexity of the R-D optimized mode decision, thereby accelerating the entire encoder by about 17% to 25% with almost no degradation in the R-D performance.