Can you "read tongue movements"?

Abstract Lip reading relies on visible articulators to ease audiovisual speech understanding. However, lips and face alone provide very incomplete phonetic information: the tongue, t hat is generally not entirely seen, carries an important p art of the articulatory information not accessible through lip reading . The question was thus whether the direct and full v ision of the tongue allows tongue reading . We have therefore generated a set of audiovisual VCV stimuli by contro lling an audiovisual talking head that can display all speec h articulators, including tongue, in an augmented speech mode, from articulators movements tracked on a speaker. T hese stimuli have been played to subjects in a series of audiovisual perception tests in various presentation conditions (audio signal alone, audiovisual signal with profile cutaw ay display with or without tongue, complete face), at various Signal-to-Noise Ratios. The results show a given implicit effe ct of tongue reading learning, a preference for the more ecological rendering of the complete face in comparison with t he cutaway presentation, a predominance of lip reading over tongue reading, but the capability of tongue readin g to take over when the audio signal is strongly degraded or absent. We conclude that these tongue reading capabilities cou ld be used for applications in the domain of speech therapy fo r speech retarded children, perception and production rehabi litation of hearing impaired children, and pronunciation traini ng for second language learners. Index Terms : Lip reading, tongue reading, audiovisual speech perception, audiovisual talking head, hearin g losses, augmented speech.