A novel rate-based macroblock classification for fast mode decision in H.264 video coding

In this paper, a novel rate-based macroblock classification is proposed for fast mode decision in H.264 video coding standard. The main idea is to classify each macroblock into simple motion or complex motion contents based on the Inter16×16 mode's residue block bit-rate and then according to the classification different mode searching orders with distinct early termination schemes are employed. This new algorithm is very simple for both hardware and software implementations without extra computational module. To speedup the intra mode decision, a new fast Intra 4×4 mode selection algorithm is also proposed by choosing most likely modes using the low complexity SATD cost as screening function. It is demonstrated by experimental results that the proposed rate-based fast algorithm can reduce 47% to 65% of the H.264 total encoding time with negligible degradation in the rate-distortion performance. While the rate-based algorithm combined with the fast intra mode selection method could further speedup 5% to 10% of the encoding time with only little rate-distortion degradation.