Fast Inter Mode Decision Algorithm Based on DCT Transform and Residual for H.264/AVC

H.264/AVC achieves extremely higher coding efficiency compared to any other previous standards. Because H.264/AVC adopts Rate-Distortion Optimization (RDO) technology with full mode search in inter prediction, the computational complexity is high accordingly. The computational complexity is so high that it’s not fit for real time encoding and hardware design. So we propose a fast inter prediction algorithm based on DCT transform and residual. In video sequences, the percent of skip mode in all the modes is very high, so we determine whether skip mode is the best mode in advance. Then we distinguish the other inter prediction modes into two classification: big prediction modes(16×16,16×8,8×16) and small prediction modes(8×8,8×4,4×8,4×4). We only choose big prediction modes or small prediction modes to encode through the comparison of residual and threshold. The experimental result shows our algorithm can reduce the encoding time obviously with little change of PSNR and Bit-rate.