Algorithmic optimization of H.264/AVC encoder

Several platform independent optimizations for a baseline profile H.264/AVC encoder are described. The optimizations include adaptive diamond pattern based motion estimation, fast sub-pel motion vector refinement and heuristic intra prediction. In addition, loop unrolling, early out thresholds and adaptive inverse transforms are used. An experimental complexity analysis is presented studying effect of optimizations on the encoding frame rate on the AMD Athlon processor. Trade-offs in rate-distortion performance are also measured. Compared to a public reference encoder, speed-ups of 4-8 have been obtained with 0.6-0.8 dB loss in image quality. In practice, our software only H.264 encoder achieves an encoding rate of 86 QCIF frames/s that is well above real-time limits.