Early Termination for Residual Quadtree Decision-Making in HEVC

Abstract The progressive high efficiency video coding (HEVC) standard is based on a quadtree (QT) coding structure. The optimal residual quadtree (RQT) was selected for a given intra- and inter-prediction residual block by comparing the rate-distortion (R-D) cost function via all possible transform unit (TU) partitions recursively. However, zero-quantized blocks (ZQBs) are common after discrete cosine transform (DCT) and quantization (Q) due to the small values of prediction blocks. Therefore, when a large TU has negligible prediction residuals, the TU can be terminated early at the current depth of the RQT. This study proposes the use of ZQB detection techniques to accelerate RQT decision-making. In HEVC, RQT decision-making comprises a TU transform and TU split functions. The proposed method mathematically analyses DCT and Q processes, deriving two sufficient conditions to reduce the computational complexity of TU transform and TU split computations. Experimental results demonstrate that the proposed method is capable of reducing the number of computations associated with inter prediction RQT decisions by an average of 48.85% (low delay, high efficiency), 45.36% (low delay, low complexity), 50.96% (random access, high efficiency) and 48.33% (random access, low complexity), while retaining the encoding performance of the original HEVC encoder.