Auditory Support for Situation Awareness in Video Surveillance

We introduce a parameter mapping sonification to support situational awareness of surveillance operators during their task of monitoring video data. The presented auditory display produces a continuous ambient soundscape reflecting the changes in video data. For this purpose, we use low-level computer vision techniques, such as optical-flow extraction and background subtraction, and rely on the capabilities of the human auditory system for high-level recognition. Special focus is put on the mapping between video features and sound parameters. We optimize this mapping to provide a good interpretability of the sound pattern, as well as an aesthetic non-obtrusive sonification: precision of the conveyed information, psychoacoustic capabilities of the auditory system, and aesthetical guidelines of sound design are considered by optimally balancing the mapping parameters using gradient descent. A user study evaluates the capabilities and limitations of the presented sonification, as well as its applicability to supporting situational awareness in surveillance scenarios.

[1]  B. Lindblom,et al.  Modeling the judgment of vowel quality differences. , 1981, The Journal of the Acoustical Society of America.

[2]  Ronald A. Rensink,et al.  TO SEE OR NOT TO SEE: The Need for Attention to Perceive Changes in Scenes , 1997 .

[3]  Mary L. Cummings,et al.  Audio Decision Support for Supervisory Control of Unmanned Vehicles Literature Review , 2006 .

[4]  F. Pulvermüller,et al.  Language outside the focus of attention: The mismatch negativity as a tool for studying higher cognitive processes , 2006, Progress in Neurobiology.

[5]  D. Benson.  Music: A Mathematical Offering , 2006 .

[6]  Michael J. Brooks,et al.  Issues in Automated Visual Surveillance , 2003, DICTA.

[7]  R. Plomp,et al.  Tonal consonance and critical bandwidth. , 1965, The Journal of the Acoustical Society of America.

[8]  John G. Neuhoff,et al.  Parameter Mapping Sonification , 2011 .

[9]  Jean-Marc Pelletier.  Sonified Motion Flow Fields as a Means of Musical Expression , 2008, NIME.

[10]  Søren H. Nielsen,et al.  Evaluation of Different Loudness Models with Music and Speech Material , 2004 .

[11]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[12]  A. Mack.  Inattentional Blindness , 2003 .

[13]  R. Proctor,et al.  Attention: Theory and Practice , 2003 .

[14]  Hina Uttam Keval,et al.  Effective design, configuration, and use of digital CCTV , 2009 .

[15]  E. Zwicker,et al.  Analytical expressions for critical‐band rate and critical bandwidth as a function of frequency , 1980 .

[16]  Yôiti Suzuki,et al.  Equal-loudness-level contours for pure tones. , 2004, The Journal of the Acoustical Society of America.

[17]  R. Näätänen,et al.  The mismatch negativity (MMN) in basic research of central auditory processing: A review , 2007, Clinical Neurophysiology.

[18]  Nahum Kiryati,et al.  Real-time abnormal motion detection in surveillance video , 2008, 2008 19th International Conference on Pattern Recognition.

[19]  Daniel Weiskopf,et al.  Interactive Auditory Display to Support Situational Awareness in Video Surveillance , 2011 .

[20]  Keith Andrews.  Evaluation Comes in Many Guises , 2008 .

[21]  Peter B. L. Meijer,et al.  An experimental system for auditory image representations , 1992, IEEE Transactions on Biomedical Engineering.