Predictive Lagrange Multiplier Selection for Perceptual Rate-Distortion Optimization

The rate-distortion optimization (RDO) framework allows a tradeoff between rate and distortion. However, objective distortion metrics such as mean square error traditionally used in the framework are poorly correlated with human perception. To address this issue, we incorporate the structural similarity index as the quality metric in the framework. A predictive Lagrange multiplier selection method is developed to resolve the chicken-and-egg dilemma of perceptual-based RDO. The resulting perceptual-based RDO framework is applied to H.264 mode decision. Given a perceptual quality level, up to 20% bit-rate reduction over the JM reference software is achieved. Subjective evaluation further confirms that, at the same bit-rate, the proposed RDO preserves more image details and prevents blocky artifact better than the traditional RDO.