Mode-dependent transforms based on elliptical model for high efficiency video coding

High efficiency video coding (HEVC) defines 35 prediction modes in its intra prediction stage to signal the direction information of residual blocks. Traditionally, separable two-dimension (2-D) transforms (integer DCT and DST) are utilized in a similar manner as in the previous H.264/AVC standards. However, such 2-D transforms cannot yield the best energy compaction for a 2-D directional source where the dominating directional information is other than the horizontal or vertical one. In order to overcome this drawback, we build an elliptical model with directionality and design some non-separable transforms based on the Karhunen-Loeve transform in this paper. Specifically, we derive a non-separable transform in closed-form for each intra-prediction mode and replace the default transform in HEVC. Simulation results reveal that 1.7% and 2.0% on average and up to 7.7% and 8.1% BD-rate reduction can be achieved for luma and chroma component, respectively. In the meantime, the test results show that both the encoding time and decoding time increase only about 5%.