Stylized ambient displays of digital media collections

The falling cost of digital cameras and camcorders has encouraged the creation of massive collections of personal digital media. However, once captured, this media is infrequently accessed and often lies dormant on users' PCs. We present a system to breathe life into home digital media collections, drawing upon artistic stylization to create a ''Digital Ambient Display'' that automatically selects, stylizes and transitions between digital contents in a semantically meaningful sequence. We present a novel algorithm based on multi-label graph cut for segmenting video into temporally coherent region maps. These maps are used to both stylize video into cartoons and paintings, and measure visual similarity between frames for smooth sequence transitions. The system automatically structures the media collection into a hierarchical representation based on visual content and semantics. Graph optimization is applied to adaptively sequence content for display in a coarse-to-fine manner, driven by user attention level (detected in real-time by a webcam). Our system is deployed on embedded hardware in the form of a compact digital photo frame. We demonstrate coherent segmentation and stylization over a variety of home videos and photos. We evaluate our media sequencing algorithm via a small-scale user study, indicating that our adaptive display conveys a more compelling media consumption experience than simple linear ''slide-shows''.

[1]  Mohamed Abdel-Mottaleb,et al.  Image browsing using hierarchical clustering , 1999, Proceedings IEEE International Symposium on Computers and Communications (Cat. No.PR00250).

[2]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[3]  Goldberg,et al.  Genetic algorithms , 1993, Robust Control Systems with Genetic Algorithms.

[4]  Guillermo Sapiro,et al.  Video SnapCut: robust video object cutout using localized classifiers , 2009, ACM Trans. Graph..

[5]  Jürgen Döllner,et al.  Image and Video Abstraction by Anisotropic Kuwahara Filtering , 2009, Comput. Graph. Forum.

[6]  Gareth Funka-Lea,et al.  Graph Cuts and Efficient N-D Image Segmentation , 2006, International Journal of Computer Vision.

[7]  Scott Cohen,et al.  LIVEcut: Learning-based interactive video segmentation by evaluation of multiple propagated cues , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[8]  Gerald Schaefer,et al.  A next generation browsing environment for large image repositories , 2010, Multimedia Tools and Applications.

[9]  Vladimir Kolmogorov,et al.  An experimental comparison of min-cut/max- flow algorithms for energy minimization in vision , 2001, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[11]  Bruno Lévy,et al.  Geometry-aware direction field processing , 2009, TOGS.

[12]  Andrew Zisserman,et al.  Video Google: a text retrieval approach to object matching in videos , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[13]  Paul A. Viola,et al.  Robust Real-time Object Detection , 2001 .

[14]  Ken Perlin,et al.  Painterly rendering for video and interaction , 2000, NPAR '00.

[15]  Matthijs C. Dorst Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[16]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[17]  Tinghuai Wang,et al.  Video stylization for digital ambient displays of home movies , 2010, NPAR.

[18]  Patrick Pérez,et al.  Interactive Image Segmentation Using an Adaptive GMMRF Model , 2004, ECCV.

[19]  Lucas Kovar,et al.  Motion graphs , 2002, SIGGRAPH Classes.

[20]  Aaron Hertzmann,et al.  Painterly rendering with curved brush strokes of multiple sizes , 1998, SIGGRAPH.

[21]  Jürgen Döllner,et al.  Image Abstraction by Structure Adaptive Filtering , 2008, TPCG.

[22]  John C. Dalton,et al.  Hierarchical browsing and search of large image databases , 2000, IEEE Trans. Image Process..

[23]  Kerry Rodden,et al.  Does organisation by similarity assist image browsing? , 2001, CHI.

[24]  Holger Winnemöller,et al.  Real-time video abstraction , 2006, SIGGRAPH 2006.

[25]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[26]  Michael J. Black,et al.  A framework for the robust estimation of optical flow , 1993, 1993 (4th) International Conference on Computer Vision.

[27]  Douglas DeCarlo,et al.  Stylization and abstraction of photographs , 2002, ACM Trans. Graph..

[28]  Peter Litwinowicz,et al.  Processing images and video for an impressionist effect , 1997, SIGGRAPH.

[29]  Seungyong Lee,et al.  Flow-Based Image Abstraction , 2009, IEEE Transactions on Visualization and Computer Graphics.

[30]  Roelof van Zwol,et al.  Flickr tag recommendation based on collective knowledge , 2008, WWW.

[31]  Shiri Gordon,et al.  Unsupervised image-set clustering using an information theoretic framework , 2006, IEEE Transactions on Image Processing.

[32]  Scott P. Robertson,et al.  Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , 1991 .

[33]  Benjamin B. Bederson,et al.  Does zooming improve image browsing? , 1999, DL '99.

[34]  Tom Appolloni,et al.  Proceedings of the 29th annual conference on Computer graphics and interactive techniques , 2002, SIGGRAPH.

[35]  Koen E. A. van de Sande,et al.  Evaluating Color Descriptors for Object and Scene Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[38]  Irfan A. Essa,et al.  Image and video based painterly animation , 2004, NPAR '04.

[39]  David Salesin,et al.  Keyframe-based tracking for rotoscoping and animation , 2004, ACM Trans. Graph..

[40]  Taku Komura,et al.  Topology matching for fully automatic similarity estimation of 3D shapes , 2001, SIGGRAPH.

[41]  Barbara J. Meier Painterly rendering for animation , 1996, SIGGRAPH.

[42]  Michal Perdoch,et al.  Efficient sequential correspondence selection by cosegmentation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  John P. Collomosse,et al.  Stroke surfaces: temporally coherent artistic animations from video , 2005, IEEE Transactions on Visualization and Computer Graphics.

[44]  Michael Cohen,et al.  Video tooning , 2004, SIGGRAPH 2004.

[45]  Richard Szeliski,et al.  Video textures , 2000, SIGGRAPH.

[46]  Sylvain Paris,et al.  Edge-Preserving Smoothing and Mean-Shift Segmentation of Video Streams , 2008, ECCV.

[47]  Nicole Arksey Exploring the Design Space for Concurrent Use of Personal and Large Displays for In-Home Collaboration , 2007 .

[48]  Delbert Dueck,et al.  Clustering by Passing Messages Between Data Points , 2007, Science.

[49]  Pushmeet Kohli,et al.  Reduce, reuse & recycle: Efficiently solving multi-label MRFs , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[50]  Dorin Comaniciu,et al.  Mean Shift: A Robust Approach Toward Feature Space Analysis , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[51]  John Philip Collomosse,et al.  Higher level techniques for the artistic rendering of images and video , 2004 .

[52]  Edward A. Fox,et al.  Proceedings of the Fourth ACM conference on Digital Libraries, August 11-14, 1999, Berkeley, CA, USA , 1999 .

[53]  Michael F. Cohen,et al.  GradientShop: A gradient-domain optimization framework for image and video filtering , 2010, TOGS.

[54]  Leonidas J. Guibas,et al.  Image webs: Computing and exploiting connectivity in image collections , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[55]  Tinghuai Wang,et al.  An Evolutionary Approach to Automatic Video Editing , 2009, 2009 Conference for Visual Media Production.

[56]  Wei You,et al.  Studying vision-based multiple-user interaction with in-home large displays , 2008, HCC '08.