Boundary-aided Extreme Value Detection based pre-processing algorithm for H.264/AVC fast intra mode prediction

The mode decision in the intra prediction of an H.264/AVC encoder requires complex computations and a significant amount of time to select the best mode that achieves the minimum rate-distortion (RD). The complex computations for the mode decision cause difficulty in real-time applications, especially for software-based H.264/AVC encoders. This study proposes an efficient fast algorithm called Boundary-aided Extreme Value Detection (BEVD) to predict the best direction mode, excluding the DC mode, for fast intra-mode decision. The BEVD-based edge detection can predict luma-4×4, luma-16×16, and chroma-8×8 modes effectively. The first step involves using the pre-processing mode selection algorithm to find the primary mode that can be selected for fast prediction. The second step requires applying the selected fewer high-potential candidate modes to calculate the RD cost for the mode decision. The encoding time is largely reduced, and similar video quality is also maintained. Simulation results show that the proposed BEVD method reduces encoding time by 63 %, and requires a bit-rate increase of approximately 1.7 %, and a decrease in peak signal-to-noise ratio (PSNR) by approximately 0.06 dB in QCIF and CIF sequences, compared with the H.264/AVC JM 14.2 software. The proposed method achieves less PSNR degradation and bit-rate increase compared to previous methods with more encoding time reduction.