On joint optimization of motion compensation, quantization and baseline entropy coding in H.264 with complete decoder compatibility

The paper presents a framework for jointly designing motion compensation, quantization and entropy coding in a hybrid video coding structure to minimize a rate distortion cost. Given motion compensation, a soft decision-based quantization algorithm is first designed to reduce the rate distortion cost by adapting quantization outputs to the baseline entropy coding method in the newest standard H.264. Motion compensation is then optimized by searching for a prediction to reduce the rate distortion cost further based on given quantization outputs. By alternating these two steps, an iterative method is then proposed. The proposed algorithms have been implemented based on the reference encoder of H.264 with complete baseline decoder compatibility. Comparative studies show that the baseline-based iterative optimization method achieves coding performance comparable, or sometimes superior, to that afforded by the main profile encoder.