Rate distortion optimized quantization for H.264/AVC based on dynamic programming

In this paper, an optimal quantization decision algorithm in the rate distortion sense for H.264/AVC1 with dynamic programming is proposed, in which a trellis is constructed for the 4x4 block quantization and adaptive entropy encoding process. Dynamic programming is employed to go through the trellis and search for the best path with minimum rate distortion cost in linear time. The quantization decisions for all coefficients in one block are made simultaneously in our approach. Thorough experiments are conducted on various sequences with different coding complexity for a wide range of bit-rates. The results show that the performance gain increases as the reproduction PSNR gets higher. For the sequences with high coding complexity, up to approximate 1 dB gain over independent quantization is observed for high PSNR encoding, e.g. higher than 45dB. In other cases, obvious gain can also be achieved. At the same time, since the state space is compact, the complexity increase is acceptable for software implementation.