Intra mode dependent quantization error estimation of each DCT coefficient in H.264/AVC

H.264/AVC employs intra prediction to reduce spatial redundancy between neighboring blocks. Different directional prediction modes are used to cater diversified video content. Although it achieves quite high coding efficiency, it is desirable to establish a proper theoretical quantization error model under different prediction modes, since this allows us to explain the behavior of existing codecs and to design better ones. Actually, residue after different intra prediction modes exhibits different characteristics in frequency domain. In this paper, an intra mode dependent quantization error estimation is presented. For a complete analysis, we investigate not only the coding distortion in current JM reference software, but also its effect on intra mode dependent residue. Based on the mode dependent residue characteristics, a mode dependent quantization error estimation for each frequency position is proposed. Simulation results show that the proposed model can estimate the mode dependent quantization error with high accuracy. Furthermore, the estimation accuracy remains high for various sequences and QP.