Real-time video abstraction

We present an automatic, real-time video and image abstraction framework that abstracts imagery by modifying the contrast of visually important features, namely luminance and color opponency. We reduce contrast in low-contrast regions using an approximation to anisotropic diffusion, and artificially increase contrast in higher contrast regions with difference-of-Gaussian edges. The abstraction step is extensible and allows for artistic or data-driven control. Abstracted images can optionally be stylized using soft color quantization to create cartoon-like effects with good temporal coherence. Our framework design is highly parallel, allowing for a GPU-based, real-time implementation. We evaluate the effectiveness of our abstraction framework with a user-study and find that participants are faster at naming abstracted faces of known persons compared to photographs. Participants are also better at remembering abstracted images of arbitrary scenes in a memory task.

[1]  Gunther Wyszecki,et al.  Color Science: Concepts and Methods, Quantitative Data and Formulae, 2nd Edition , 2000 .

[2]  Takafumi Saito,et al.  Comprehensible rendering of 3-D shapes , 1990, SIGGRAPH.

[3]  Erik Reinhard,et al.  Human facial illustrations: Creation and psychophysical evaluation , 2004, TOGS.

[4]  David Salesin,et al.  Computer-generated pen-and-ink illustration , 1994, SIGGRAPH.

[5]  C. Koch,et al.  Computational modelling of visual attention , 2001, Nature Reviews Neuroscience.

[6]  Joost van de Weijer,et al.  On the Equivalence of Local-Mode Finding, Robust Estimation and Mean-Shift Analysis as Used in Early Vision Tasks , 2002, ICPR.

[7]  Jitendra Malik,et al.  Scale-Space and Edge Detection Using Anisotropic Diffusion , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Aaron Hertzmann,et al.  Paint by relaxation , 2001, Proceedings. Computer Graphics International 2001.

[9]  Craig Gotsman,et al.  Enhancement by image-dependent warping , 1999, IEEE Trans. Image Process..

[10]  John P. Collomosse,et al.  Stroke surfaces: temporally coherent artistic animations from video , 2005, IEEE Transactions on Visualization and Computer Graphics.

[11]  Claudio M. Privitera,et al.  Algorithms for Defining Visual Regions-of-Interest: Comparison with Eye Fixations , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Dirk Bartz,et al.  Stylized augmented reality for improved immersion , 2005, IEEE Proceedings. VR 2005. Virtual Reality, 2005..

[13]  Sarah V. Stevenage,et al.  Can caricatures really produce distinctiveness effects , 1995 .

[14]  Douglas DeCarlo,et al.  Stylization and abstraction of photographs , 2002, ACM Trans. Graph..

[15]  D Marr,et al.  Theory of edge detection , 1979, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[16]  H. Barlow.  Vision Science: Photons to Phenomenology by Stephen E. Palmer , 2000, Trends in Cognitive Sciences.

[17]  Harry Shum,et al.  Video tooning , 2004, ACM Trans. Graph..

[18]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[19]  Lucas J. van Vliet,et al.  Separable bilateral filtering for fast video preprocessing , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[20]  Douglas DeCarlo,et al.  Visual interest and NPR: an evaluation and manifesto , 2004, NPAR '04.

[21]  Ramesh Raskar,et al.  Non-photorealistic camera: depth edge detection and stylized rendering using multi-flash imaging , 2004, ACM Trans. Graph..

[22]  James H. Elder,et al.  Are Edges Incomplete? , 1999, International Journal of Computer Vision.

[23]  Dorin Comaniciu,et al.  A common framework for nonlinear diffusion, adaptive smoothing, bilateral filtering and mean shift , 2004, Image Vis. Comput..

[24]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.