An audio-visual distance for audio-visual speech vector quantization

Speech is both an acoustic and a visual signal, and there exists some complementarity and redundancy between the two modalities. In the speech coding domain, it is of great interest to use this redundancy to improve speech coder performance. In this paper, we consider some audio and video joint coding process based on an audio-visual vector quantization. The method is shown to exploit quite well the audio-visual redundancy as it can reduce the bit rate while decreasing the quantization error. A notion of audio-visual distance has to be introduced and adapted to the different nature of the data. It is defined from an existing audio distance and a new visual distance, which is particularly focussed.