New vector transform for image coding

Vector quantization (VQ) always outperforms scalar quantization. Recently, vector transform coding (VTC) has been introduced to better take advantage of signal processing for vector quantization and shown to achieve better performance in image coding. How much performance advantage in terms of rate-distortion can a vector transform coding scheme gain over the other coding schemes? What is the optimal vector transform (VT) with complexity constraint on VQ? These are the questions we try to answer in this paper. Based on the results from high-resolution or asymptotic (in rate) quantization theory, we obtain a general rate- distortion formula for signal processing combined with vector quantization for first-order Gaussian-Markov source. We prove that VTC indeed has better performance than other existing coding schemes with the same or less complexity based on the rate-distortion measurement. A new mirror-sampling based vector transform which only involves additions and subtractions is proposed. For high rate case, we show that the new VTC scheme achieves the optimal performance under the complexity constraint. A 2D version of the new vector transform is applied to image coding, and the results show that the new vector transform consistently outperforms the subsampling-based vector transform.