A User Experience Model for Home Video Summarization

In this paper, we propose a novel system for automatically summarizing home videos based on a user experience model. The user experience model takes account of user's spontaneous behaviors when viewing videos. Based on users' reaction when viewing videos, we can construct a systematic framework to automate video summarization. In this work, we analyze the variations of viewer's eye movement and facial expression when he or she watching the raw home video. We transform these behaviors into the clues of determining the important part of each video shot. With the aids of music analysis, the developed system automatically generates a music video (MV) style summarized home videos. Experiments show that this new type of editing mechanism can effectively generate home video summaries and can largely reduce the efforts of manual summarization.

[1]  Wa James Tam,et al.  Processing image sequences based on eye movements , 1994, Electronic Imaging.

[2]  Filip Germeys,et al.  The psychology of film: perceiving beyond the cut , 2007, Psychological research.

[3]  Yi-Ping Hung,et al.  Analyzing Facial Expression by Fusing Manifolds , 2007, ACCV.

[4]  Wei-Lun Chang,et al.  Aesthetics-Based Automatic Home Video Skimming System , 2008, MMM.

[5]  Adrian F. Clark,et al.  An algorithm for face and facial-feature location based on grey-scale information and facial geometry , 1999 .

[6]  Alan Hanjalic.  Multimodal approach to measuring excitement in video , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[7]  S. Liversedge,et al.  Saccadic eye movements and cognition , 2000, Trends in Cognitive Sciences.

[8]  Gang Luo,et al.  Eye movements while watching video: comparisons across viewer groups , 2004 .

[9]  Tao Mei,et al.  Modeling and Mining of Users' Capture Intention for Home Videos , 2007, IEEE Transactions on Multimedia.

[10]  C. Wallraven,et al.  Processing of facial identity and expression: a psychophysical, physiological, and computational perspective. , 2006, Progress in brain research.

[11]  R. Klein,et al.  Does Oculomotor Readiness Mediate Cognitive Control of Visual-Attention - Revisited , 1994 .

[12]  Patrick McGrath,et al.  Editing digital video , 2003 .

[13]  Gael Chandler,et al.  Cut by Cut: Editing Your Film or Video , 2004 .

[14]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[15]  V. Tosi,et al.  Scanning eye movements made when viewing film: preliminary observations. , 1997, The International journal of neuroscience.

[16]  P. Ekman.  Unmasking The Face , 1975 .

[17]  Anindya Sarkar,et al.  Feature fusion and redundancy pruning for rush video summarization , 2007, TVS '07.

[18]  Alex Pentland,et al.  Coding, Analysis, Interpretation, and Recognition of Facial Expressions , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Svetha Venkatesh,et al.  Computational Media Aesthetics: Finding Meaning Beautiful , 2001, IEEE Multim..

[20]  H. Zettl.  Sight, Sound, Motion: Applied Media Aesthetics , 1973 .

[21]  Gwen Littlewort,et al.  Recognizing facial expression: machine learning and application to spontaneous behavior , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[22]  Mohan S. Kankanhalli,et al.  Pivot Vector Space Approach for Audio-Video Mixing , 2003, IEEE Multim..

[23]  Azriel Rosenfeld,et al.  Eye detection in a face image using linear and nonlinear filters , 2001, Pattern Recognit..

[24]  Eli Peli,et al.  Where people look when watching movies: Do all viewers look at the same place? , 2007, Comput. Biol. Medicine.

[25]  Lie Lu,et al.  A generic framework of user attention model and its application in video summarization , 2005, IEEE Trans. Multim..

[26]  Patrick McGrath,et al.  Editing Digital Video : The Complete Creative and Technical Guide , 2002 .