Using Speech in Visual Object Recognition

Automatic understanding of multi-modal input is the central topic in modern human Computer interfaces. But the basic questions about how the interpretations provided by different modalities can be connected in a universal and robust manner is still an open problem. The most intuitive input modalities, speech perception and vision, can only be correlated on a qualitative content based interpretation level. But, due to vague meanings and erroneous processing results this is extremely difficult to accomplish. A simple frame based integration scheme filling appropriate slots with new analysis results will fail when ambiguous or contradictory information appears. In this paper we propose a new probabilistic framework to overcome these drawbacks. The integration model is built up from data collected in labeled test sets and psycholinguistic experiments. Thereby, the correspondence problem is solved in a very robust and universal manner. In particular, we will show that erroneous visual interpretations can be corrected by a joint analysis of visual and speech input data.