Improved H.264 rate control by enhanced MAD-based frame complexity prediction

Abstract This paper presents a revised rate control scheme based on an improved frame complexity measure. Rate control adopted by both MPEG-4 VM18 and H.264/AVC use a quadratic rate–distortion (R–D) model that determines quantization parameters (QPs). Classical quadratic R–D model is suitable for MPEG-4 but it performs poorly for H.264/AVC because one of the important parameters, mean absolute difference (MAD), is predicted through a linear model, whereas the MAD used in MPEG-4 VM18 is the actual MAD. Inaccurately predicted MAD results in wrong QP and consequently degrades rate–distortion optimization (RDO) performance in H.264. To overcome the limitation of the existing rate control schemes, we introduce an enhanced linear model for predicting MAD, utilizing some knowledge of current frame complexity. Moreover, we propose a more accurate frame complexity measure, namely, normalized MAD, to replace the current use of MAD parameter. Normalized MAD has a stronger correlation with optimally allocated bits than that of the predicted MAD. To minimize video quality variations, we also propose a novel long-term QP limiter (LTQPL). Finally, a dynamic bit allocation scheme among basic units is implemented. Extensive simulation results show that our method, with inexpensive computational complexity added, improves the average peak signal-to-noise ratio (PSNR) and reduces video quality variations considerably.