Mode dependent DCT/DST for intra prediction in block-based image/video coding

In this paper, we present a mode-dependent transform scheme that applies either the conventional DCT or type-7 DST for all the video-coding intra-prediction modes: vertical, horizontal or oblique. Our approach is applicable to any block-based intra prediction scheme in a codec, that employs transforms along the horizontal and vertical direction separably. Previously Han, Saxena & Rose showed that for the intra-predicted residuals of horizontal and vertical modes, the DST is the optimal transform with performance close to KLT. Here we prove that this is indeed the case for the other oblique modes. The choice of using DCT/DST is based on intra-prediction modes, and requires no additional signaling information or Rate-Distortion search. Simulations are conducted for the DCT/DST algorithm in TMuC 0.9, the reference software for the ongoing HEVC standardization. Our results show that the DCT/DST scheme provides significant BD-Rate improvement over the DCT for intra prediction in video sequences.