Comparing spatial masking modelling in just noticeable distortion controlled H.264/AVC video coding

This paper studies the integration of a just noticeable distortion model in a H.264/AVC standard codec to improve the final rate-distortion performance. Three masking aspects related to lossy transform coding and natural video contents are considered: frequency band decomposition, luminance component variations and pattern masking. For the latter aspect, three alternative models are considered, namely the Foley-Boynton, Foley-Boynton adaptive and Wei-Ngan models. Their performance, measured for high definition video content, and reported in terms of bitrate improvement and objective quality loss, reveals that the Foley-Boynton and its adaptive version provide the best performance with up to 35.6 % bitrate reduction at the cost of at most 1.4 % objective quality loss.