Rate-distortion optimization for arbitrarily-shaped object coding

The trade-off between rate and distortion suggests two approaches for the rate-distortion optimization: (1) reduce the rate at a smaller increase in distortion (pushing the rate-distortion curve to the left) and (2) reduce the distortion at a smaller increase in rate (pushing the rate-distortion curve downward). In general, only the first approach is applicable for ordinary fully-defined blocks of square size, which is the case in all existing image/video coding standards other than MPEG-4. The authors show that for arbitrarily-shaped video objects, the inner object blocks can be optimized in the same way; while the boundary blocks (of arbitrary shapes), if padded properly, can be optimized using not only the first approach but also the second approach as well. To circumvent the high computational complexity, fully embedded pattern tables are designed so that the most computationally expensive part can be replaced by relatively simple table searching operations. Extensive simulation have been done and the results are found to support the conclusion drawn