Efficient mode selection with extreme value detection based pre-processing algorithm for H.264/AVC fast intra mode decision

The mode decision in the intra prediction of a H.264/AVC encoder requires complicated computations, and spends much time to select the best mode that achieves the minimum rate-distortion (RD). The complicated computations for the mode decision cause the difficulty in real-time applications, especially for software based encoders. This study creates an efficient fast algorithm, which is called Extreme Value Detection (EVD), to predict the best direction mode except for the DC mode for fast intra mode decision. The EVD based edge detection predicts luma-4×4, luma-16×16, and chroma-8×8 modes effectively. At the first step, we use the pre-processing mode selection algorithm to find the primary mode which is selected for fast prediction. At the second step, the selected fewer high-potential candidate modes are applied to calculate the RD cost for the mode decision. This method reduces encoding time effectively, and meanwhile also maintains the same video quality. Simulation results show that the proposed EVD method reduces the encoding time by 63%, and requires bit-rate increase about 2.6% and peak signal-to-noise ratio (PSNR) decrease about 0.08 dB in QCIF and CIF sequences, compared with the H.264/AVC JM 14.2 software. This method achieves less PSNR degradation and bit-rate increase than the previous methods with more encoding time reduction.