Exploiting Audio-Visual Correlation by Means of Gaze Tracking

This paper presents a novel means for increasing audio-visual correlation analysis reliability. This is done based on gaze tracking technology engineered at the Multimedia Systems Department of the Gdansk University of Technology, Poland. In the paper, the past history and current research in the area of audio-visual perception analysis are shortly reviewed. Then the methodology employing gaze tracking is presented along with the results of audio-visual experiments performed. It is found that the methodology presented can be used to study audio-video correlation. This is confirmed by a sufficiently high correlation between subjective assessment and objective test results. In this study Pearson‟s correlation coefficient computed for the subjective responses (MOS) and the objective measure equals 0.63 (at the 95% confidence level).

[1]  Kaoru Watanabe,et al.  A Method of 3-D Sound Image Localization using Loudspeaker Arrays , 2003 .

[2]  Sridha Sridharan,et al.  Visual attention based ROI maps from gaze tracking data , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[3]  Bozena Kostek,et al.  An new method of audio-visual correlation analysis , 2009, 2009 International Multiconference on Computer Science and Information Technology.

[4]  Søren Bech,et al.  Interaction Between Audio-Visual Factors in a Home Theater System: Definition of Subjective Attributes , 1995 .

[5]  Andrzej Czyzewski,et al.  Making Surround Audio Considering Image Proximity Effect , 2002 .

[6]  Andrzej Czyzewski,et al.  Determination of Influence of Visual Cues on Perception of Spatial Sound , 2001 .

[7]  G. J. Thomas.  Experimental study of the influence of vision on sound localization. , 1941 .

[8]  Francis Rumsey,et al.  Can Playing a Computer Game Affect Perception of Audio-Visual Synchrony? , 2004 .

[9]  Andrzej Czyzewski,et al.  Determining Influence of Visual Cues on the Perception of Surround Sound Using Soft Computing , 2000, Rough Sets and Current Trends in Computing.

[10]  Setsu Komiyama,et al.  Subjective Evaluation of Multi-Channel Stereophony for HDTV , 1987, IEEE Transactions on Broadcasting.

[11]  Yoichi Sato,et al.  Recovering audio-to-video synchronization by audiovisual correlation analysis , 2008, 2008 19th International Conference on Pattern Recognition.

[12]  Bozena Kostek.  Perception-Based Data Processing in Acoustics: Applications to Music Information Retrieval and Psychophysiology , 2005, Studies in Computational Intelligence.

[13]  H. A. Witkin,et al.  Sound localization with conflicting visual and auditory cues. , 1952, Journal of experimental psychology.

[14]  Toshiyuki Gotoh,et al.  Controlling Sound-Image Localization in Stereophonic Reproduction: II , 1981 .

[15]  Sugato Chakravarty,et al.  Methodology for the subjective assessment of the quality of television pictures , 1995 .

[16]  Nina Ivanovna Dvorko,et al.  Audio-visual Perception of Video and Multimedia Programs , 2002 .

[17]  Francis Rumsey,et al.  Computer Games and Multichannel Audio Quality - The Effect of Division of Attention between Auditory and Visual Modalities , 2003 .

[18]  Andrzej Czyzewski,et al.  Concentration Tests - An Application of Gaze Tracker to Concentration Exercises , 2009, CSEDU.

[19]  A. Murat Tekalp,et al.  Audiovisual Synchronization and Fusion Using Canonical Correlation Analysis , 2007, IEEE Transactions on Multimedia.

[20]  David J. Meares.  Perceptual Attributes of Multichannel Sound , 1993 .

[21]  Søren Bech,et al.  Interaction Between Audio-Visual Factors in a Home Theater System: Experimental Results , 1995 .

[22]  Setsu Komiyama,et al.  Subjective Evaluation of Angular Displacement between Picture and Sound Directions for HDTV Sound Systems , 1989 .

[23]  Andrzej Czyzewski,et al.  Discovering the Influence of Visual Stimuli on the Perception of Surround Sound Using Genetic Algorithms , 2001 .

[24]  Daniel Thalmann,et al.  Evaluation of Gaze Tracking Technology for Social Interaction in Virtual Environments , 2004 .

[25]  Ioan Allen,et al.  Matching the Sound to the Picture , 1991 .

[26]  Fabien Ringeval,et al.  Maximising Audiovisual Correlation with Automatic Lip Tracking and Vowel Based Segmentation , 2009, COST 2101/2102 Conference.

[27]  Andrzej Czyzewski,et al.  Influence of visual cues on the perception of surround sound , 2000 .

[28]  Andrzej Czyzewski,et al.  Gaze-tracking-based audio-visual correlation analysis employing quality of experience methodology , 2010, Intell. Decis. Technol..