A sparse control model for image and video editing

It is common that users draw strokes, as control samples, to modify color, structure, or tone of a picture. We discover inherent limitation of existing methods for their implicit requirement on where and how the strokes are drawn, and present a new system that is principled on minimizing the amount of work put in user interaction. Our method automatically determines the influence of edit samples across the whole image jointly considering spatial distance, sample location, and appearance. It greatly reduces the number of samples that are needed, while allowing for a decent level of global and local manipulation of resulting effects and reducing propagation ambiguity. Our method is broadly beneficial to applications adjusting visual content.

[1]  Frédo Durand,et al.  A Fast Approximation of the Bilateral Filter Using a Signal Processing Approach , 2006, ECCV.

[2]  Andrew Adams,et al.  Fast High‐Dimensional Filtering Using the Permutohedral Lattice , 2010, Comput. Graph. Forum.

[3]  Chi-Keung Tang,et al.  KNN Matting , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Manuel Menezes de Oliveira Neto,et al.  Adaptive manifolds for real-time high-dimensional filtering , 2012, ACM Trans. Graph..

[5]  Edward H. Adelson,et al.  Eurographics Symposium on Rendering 2008 Scribbleboost: Adding Classification to Edge-aware Interpolation of Local Image and Video Adjustments , 2022 .

[6]  Jue Wang,et al.  A perceptually motivated online benchmark for image matting , 2009, CVPR.

[7]  Li Xu,et al.  Structure extraction from texture via relative total variation , 2012, ACM Trans. Graph..

[8]  Dani Lischinski,et al.  Joint bilateral upsampling , 2007, SIGGRAPH 2007.

[9]  Fabio Pellacini,et al.  AppProp: all-pairs appearance-space edit propagation , 2008, ACM Trans. Graph..

[10]  Michael F. Cohen,et al.  Optimized Color Sampling for Robust Matting , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Maneesh Agrawala,et al.  Multiscale shape and detail enhancement from multi-light image collections , 2007, SIGGRAPH 2007.

[12]  Jan Kautz,et al.  Local Laplacian filters: edge-aware image processing with a Laplacian pyramid , 2011, SIGGRAPH 2011.

[13]  Dani Lischinski,et al.  A Closed-Form Solution to Natural Image Matting , 2008 .

[14]  Ying Wu,et al.  Nonlocal matting , 2011, CVPR 2011.

[15]  M. Kass,et al.  Smoothed local histogram filters , 2010, SIGGRAPH 2010.

[16]  Markus Gross,et al.  Practical temporal consistency for image-based graphics applications , 2012, ACM Trans. Graph..

[17]  Jitendra Malik,et al.  Spectral grouping using the Nystrom method , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Shi-Min Hu,et al.  Efficient affinity-based edit propagation using K-D tree , 2009, ACM Trans. Graph..

[19]  XuLi,et al.  A sparse control model for image and video editing , 2013 .

[20]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[21]  Tien Tsin,et al.  Image Partial Blur Detection and Classification , 2013 .

[22]  Frédo Durand,et al.  Edge-preserving multiscale image decomposition based on local extrema , 2009, ACM Trans. Graph..

[23]  Shi-Min Hu,et al.  Instant Propagation of Sparse Edits on Images and Videos , 2010, Comput. Graph. Forum.

[24]  M. Levoy,et al.  Gaussian KD-trees for fast high-dimensional filtering , 2009, SIGGRAPH 2009.

[25]  Manuel M. Oliveira,et al.  Domain transform for edge-aware image and video processing , 2011, SIGGRAPH 2011.

[26]  Frédo Durand,et al.  Decoupling algorithms from schedules for easy optimization of image processing pipelines , 2012, ACM Trans. Graph..

[27]  Baining Guo,et al.  Edit Propagation on Bidirectional Texture Functions , 2009, Comput. Graph. Forum.

[28]  Jason Lawrence,et al.  AppWand: editing measured materials using appearance-driven optimization , 2007, SIGGRAPH 2007.

[29]  Frédo Durand,et al.  Two-scale tone management for photographic look , 2006, SIGGRAPH 2006.

[30]  Zeev Farbman,et al.  Interactive local adjustment of tonal values , 2006, ACM Trans. Graph..

[31]  Frédo Durand,et al.  User-guided white balance for mixed lighting conditions , 2012, ACM Trans. Graph..

[32]  Cewu Lu,et al.  Image smoothing via L0 gradient minimization , 2011, ACM Trans. Graph..

[33]  Michael F. Cohen,et al.  An iterative optimization approach for unified image segmentation and matting , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[34]  Jiawen Chen,et al.  Real-time edge-aware image processing with the bilateral grid , 2007, SIGGRAPH 2007.

[35]  Dani Lischinski,et al.  Colorization using optimization , 2004, ACM Trans. Graph..

[36]  Raanan Fattal,et al.  Diffusion maps for edge-aware image editing , 2010, SIGGRAPH 2010.

[37]  Fabio Pellacini,et al.  AppProp: all-pairs appearance-space edit propagation , 2008, SIGGRAPH 2008.

[38]  Alexei A. Efros,et al.  Fast bilateral filtering for the display of high-dynamic-range images , 2002 .

[39]  D. Shepard.  A two-dimensional interpolation function for irregularly-spaced data , 1968, ACM National Conference.

[40]  Xiaowu Chen,et al.  Manifold preserving edit propagation , 2012, ACM Trans. Graph..

[41]  Vladlen Koltun,et al.  Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials , 2011, NIPS.

[42]  Dani Lischinski,et al.  Spectral Matting , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Zeev Farbman,et al.  Edge-preserving decompositions for multi-scale tone and detail manipulation , 2008, SIGGRAPH 2008.

[44]  Raanan Fattal,et al.  Edge-avoiding wavelets and their applications , 2009, ACM Trans. Graph..

[45]  Patrick Pérez,et al.  Geodesic image and video editing , 2010, TOGS.