Hearing by Eye: How Much Spatial Degradation can Be Tolerated?

In the McGurk effect (McGurk and MacDonald, 1976 Nature 264 746–748), illusory auditory perception is produced if the visual information from lip movements is discrepant from the auditory information from the voice. A study is reported of the tolerance of the effect to varying levels of spatial degradation (videotaped images of a speaker's face were quantised by a mosaic transform). The illusory effect systematically decreased with an increase in the coarseness of the spatial quantisation. However, even with the coarsest level (11.2 pixels/face) the illusion did not completely disappear. In addition, those participants who did not experience the illusion nevertheless showed the effects of auditory–visual interaction in their clarity ratings of the auditory stimulus. It is concluded that auditory – visual interaction in visible speech perception is based on relatively coarse-spatial-scale information.

[1]  D. Massaro,et al.  Perception of Visible Speech: Influence of Spatial Quantization , 1997, Perception.

[2]  P. Gribble,et al.  Temporal constraints on the McGurk effect , 1996, Perception & psychophysics.

[3]  H. McGurk,et al.  Visual influences on speech perception processes , 1978, Perception & psychophysics.

[4]  H. Leder.  Line Drawings of Faces Reduce Configural Processing , 1996, Perception.

[5]  Christian Benoît,et al.  Which components of the face do humans and machines best speechread , 1996 .

[6]  W. Uttal,et al.  A parametric study of face recognition when image degradations are combined. , 1997, Spatial vision.

[7]  C. Spence,et al.  Cross-modal links in exogenous covert spatial orienting between touch, audition, and vision , 1998, Perception & psychophysics.

[8]  B. Walden,et al.  Effects of training on the visual recognition of consonants. , 1977, Journal of speech and hearing research.

[9]  J. Sergent.  Microgenesis of Face Perception , 1986 .

[10]  Vicki Bruce,et al.  Facial identity and facial speech processing: Familiar faces and voices in the McGurk effect , 1995, Perception & psychophysics.

[11]  J. Driver,et al.  Audiovisual links in exogenous covert spatial orienting , 1997, Perception & psychophysics.

[12]  Q Summerfield,et al.  Use of Visual Information for Phonetic Perception , 1979, Phonetica.

[13]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[14]  E. Vatikiotis-Bateson,et al.  Eye movement of perceivers during audiovisualspeech perception , 1998, Perception & psychophysics.

[15]  B Julesz,et al.  Masking in Visual Recognition: Effects of Two-Dimensional Filtered Noise , 1973, Science.

[16]  N. M. Brooke,et al.  Analysis, synthesis, and perception of visible articulatory movements , 1983 .

[17]  D W Massaro,et al.  American Psychological Association, Inc. Evaluation and Integration of Visual and Auditory Information in Speech Perception , 2022 .

[18]  Q. Summerfield,et al.  Lipreading and audio-visual speech perception. , 1992, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[19]  Q Summerfield,et al.  Audiovisual presentation demonstrates that selective adaptation in speech perception is purely auditory , 1981, Perception & psychophysics.

[20]  J. N. Bassili.  Facial motion in the perception of faces and of emotional expression. , 1978, Journal of experimental psychology. Human perception and performance.

[21]  J R Lishman,et al.  Evidence for the View That Temporospatial Integration in Vision is Temporally Anisotropic , 1997, Perception.

[22]  D. Massaro.  Perceiving talking faces: from speech perception to a behavioral principle , 1999 .

[23]  Ashok Samal,et al.  Human Face Perception in Degraded Images , 1995, J. Vis. Commun. Image Represent..

[24]  C A Fowler,et al.  Listeners do hear sounds, not tongues. , 1996, The Journal of the Acoustical Society of America.

[25]  J. N. Bassili.  Emotion recognition: the role of facial movement and the relative importance of upper and lower areas of the face. , 1979, Journal of personality and social psychology.

[26]  T Caelli,et al.  What is Perceived When Two Images are Combined? , 1985, Perception.

[27]  Beng Hong. Goh,et al.  Effects of training. , 2001 .

[28]  H. Wallbott.  Effects of distortion of spatial and temporal resolution of video stimuli on emotion attributions , 1992 .

[29]  D. Massaro,et al.  Perceiving Talking Faces , 1995 .

[30]  T Bachmann,et al.  Microgenesis as traced by the transient paired-forms paradigm. , 1989, Acta psychologica.

[31]  H. Hughes,et al.  Global Precedence, Spatial Frequency Channels, and the Statistics of Natural Images , 1996, Journal of Cognitive Neuroscience.

[32]  Barbara Dodd,et al.  The Role of Vision in the Perception of Speech , 1977, Perception.

[33]  H. Ellis,et al.  Masking of faces by facial and non-facial stimuli , 1994 .

[34]  R. D. Easton,et al.  Perceptual dominance during lipreading , 1982, Perception & psychophysics.

[35]  T. Bachmann.  Identification of spatially quantised tachistoscopic images of faces: How many pixels does it take to carry identity? , 1991 .

[36]  H. Wallbott.  The robustness of communication of emotion via facial expression: Emotion recognition from photographs with deteriorated pictorial quality , 1991 .

[37]  Frans J. Maarse,et al.  Initial microgenetic steps in single-glance face recognition , 1984 .

[38]  I. Craw,et al.  Effects of high-pass and low-pass spatial filtering on face identification , 1996, Perception & psychophysics.

[39]  G. Sandini,et al.  The Role of High Spatial Frequencies in Face Perception , 1983, Perception.

[40]  V Bruce,et al.  The Use of Pigmentation and Shading Information in Recognising the Sex and Identities of Faces , 1994, Perception.

[41]  H. Bülthoff,et al.  Face recognition under varying poses: The role of texture and shape , 1996, Vision Research.

[42]  Neeme Kahusk,et al.  The Effects of Coarseness of Quantisation, Exposure Duration, and Selective Spatial Attention on the Perception of Spatially Quantised (‘Blocked’) Visual Images , 1997, Perception.

[43]  G. Hole,et al.  The Influence of Feature-Based Information in the Age Processing of Unfamiliar Faces , 1998, Perception.

[44]  J R Lishman,et al.  Role of coarse and fine spatial information in face and object processing. , 1996, Journal of experimental psychology. Human perception and performance.

[45]  R. Fox.  Modularity and the Motor Theory of Speech Perception , 1994 .

[46]  J. Gibson.  The Ecological Approach to Visual Perception , 1979 .

[47]  Sarah V. Stevenage,et al.  Which twin are you? A demonstration of induced categorical perception of identical twin faces , 1998 .

[48]  I. Craw,et al.  Spatial Content and Spatial Quantisation Effects in Face Recognition , 1994, Perception.