Interactive image filtering with multiple levels-of-control on mobile devices

With the continuous development of mobile graphics hardware, interactive high-quality image stylization based on nonlinear filtering is becoming feasible and increasingly used in casual creativity apps. However, these apps often only serve high-level controls to parameterize image filters and generally lack support for low-level (artistic) control, thus automating art creation rather than assisting it. This work presents a GPU-based framework that enables to parameterize image filters at three levels of control: (1) presets followed by (2) global parameter adjustments can be interactively refined by (3) complementary on-screen painting that operates within the filters' parameter spaces for local adjustments. The framework provides a modular XML-based effect scheme to effectively build complex image processing chains-using these interactive filters as building blocks---that can be efficiently processed on mobile devices. Thereby, global and local parameterizations are directed with higher-level algorithmic support to ease the interactive editing process, which is demonstrated by state-of-the-art stylization effects, such as oil-paint filtering and watercolor rendering.

[1]  Tobias Isenberg,et al.  Interactive example-based hatching , 2013, Comput. Graph..

[2]  Bruce Gooch,et al.  Interactive vector fields for painterly rendering , 2005, Graphics Interface.

[3]  Jürgen Döllner,et al.  Image Abstraction by Structure Adaptive Filtering , 2008, TPCG.

[4]  Ramzi Mahmoudi,et al.  Ieee Ipas'14: International Image Processing Applications and Systems Conference 2014 Image Processing on Mobile Devices: an Overview , 2022 .

[5]  Adam Finkelstein,et al.  Fine tone control in hardware hatching , 2002, NPAR '02.

[6]  David Salesin,et al.  Orientable textures for image-based pen-and-ink illustration , 1997, SIGGRAPH.

[7]  Kwan-Liu Ma,et al.  Interactivity is the key to expressive visualization , 2002, COMG.

[8]  HanrahanPat,et al.  Direct WYSIWYG painting and texturing on 3D shapes , 1990 .

[9]  Tobias Isenberg,et al.  Interactive NPAR: what type of tools should we create? , 2016, Expressive.

[10]  Jakob Nielsen,et al.  Chapter 2 - What Is Usability? , 1993 .

[11]  Bruce Gooch,et al.  Viewing progress in non-photorealistic rendering through Heinlein's lens , 2010, NPAR.

[12]  Jürgen Döllner,et al.  Image stylization by interactive oil paint filtering , 2016, Comput. Graph..

[13]  Matthias Trapp,et al.  Interactive multi-scale oil paint filtering on mobile devices , 2016, SIGGRAPH Posters.

[14]  Stephen DiVerdi,et al.  A Brush Stroke Synthesis Toolbox , 2013, Image and Video-Based Artistic Stylisation.

[15]  Paul L. Rosin,et al.  A benchmark image set for evaluating stylization , 2016 .

[16]  Irfan A. Essa,et al.  Image and video based painterly animation , 2004, NPAR '04.

[17]  Jun-Hai Yong,et al.  Towards Photo Watercolorization with Artistic Verisimilitude , 2014, IEEE Transactions on Visualization and Computer Graphics.

[18]  Jürgen Döllner,et al.  BeCasso: image stylization by interactive oil paint filtering on mobile devices , 2016, SIGGRAPH Appy Hour.

[19]  In Kyu Park,et al.  Design and optimization of image processing algorithms on mobile GPU , 2011, SIGGRAPH '11.

[20]  Holger Winnemöller,et al.  Real-time video abstraction , 2006, SIGGRAPH 2006.

[22]  Tomas Akenine-Möller,et al.  The State of the Art in Mobile Graphics Research , 2008, IEEE Computer Graphics and Applications.

[23]  Yu-Kun Lai,et al.  Non-photorealistic rendering with spot colour , 2013, CAE '13.

[24]  Kapil Dev Mobile Expressive Renderings: The State of the Art , 2013, IEEE Computer Graphics and Applications.

[25]  Holger Winnemöller,et al.  XDoG: An eXtended difference-of-Gaussians compendium including advanced image stylization , 2012, Comput. Graph..

[26]  Holger Winnemöller,et al.  Painting with Bob: assisted creativity for novices , 2014, UIST.

[27]  Adrien Bousseau,et al.  Interactive watercolor rendering with temporal coherence and abstraction , 2006, NPAR.

[28]  Kyunghyun Yoon,et al.  Interactive Painterly Rendering for Mobile Devices , 2015, ICEC.

[29]  Adam Finkelstein,et al.  Real-time hatching , 2001, SIGGRAPH.

[31]  William V. Baxter,et al.  Locally controllable stylized shading , 2007, ACM Trans. Graph..

[32]  Paul L. Rosin,et al.  Image and Video-Based Artistic Stylisation , 2012, Computational Imaging and Vision.

[33]  Holger Winnemöller,et al.  NPR in the Wild , 2013, Image and Video-Based Artistic Stylisation.

[34]  Stephen DiVerdi,et al.  Painting with Polygons: A Procedural Watercolor Engine , 2013, IEEE Transactions on Visualization and Computer Graphics.

[35]  Dani Lischinski,et al.  Joint bilateral upsampling , 2007, ACM Trans. Graph..

[36]  Thomas Brox,et al.  Nonlinear structure tensors , 2006, Image Vis. Comput..

[37]  B. Thomas,et al.  Stylized Depiction in Mixed Reality , 2022 .

[38]  Holger Winnemöller,et al.  Real-time video abstraction , 2006, ACM Trans. Graph..

[39]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[40]  William V. Baxter,et al.  Tweakable light and shade for cartoon animation , 2006, NPAR.

[41]  Daniel Wexler,et al.  Intelligent brush strokes , 2012, SIGGRAPH '12.

[42]  Tobias Isenberg,et al.  State of the "Art”: A Taxonomy of Artistic Stylization Techniques for Images and Video , 2013, IEEE Transactions on Visualization and Computer Graphics.