The analysis of a fractional pixels generated by a combination of linear FIR filters in a video codec

In this paper, we propose an analysis of fractional pixels which are interpolated from integer pixels using hierarchical linear FIR filters based on a differential structure of sub-pixels. In order to improve a video quality under a limited bitrate, most of video codecs employs fractional pixels (or samples) in a process of motion compensation or motion estimation. Even though the fractional pixels have been widely used in the field of video compression, it has not been so much researched to analyze a mathematical structure about the fractional pixels which are interpolated by linear FIR filters. Through the mathematical analysis, we provide a limitation of a resolution for fractional pixels for video codecs, such as h.264/AVC(Advanced video codec) or HEVC(High Efficiency Video Codec). Practically, we find a limitation of the resolution for fractional pixels is in unit of one octa-pixel (1/8-pixels) of the distance between integer pixel positions. Moreover, we find that it is not effective to employ fractional pixels beyond 1/16-pixels by the proposed analysis. The expected improvement of a compression performance using 1/8-pixels is about 35% at most, and the result of the expected improvement is very close to the previous results of research about the adoption of 1/8-pixel to video compression.