DeepProp: Extracting Deep Features from a Single Image for Edit Propagation

Edit propagation is a technique that can propagate various image edits (e.g., colorization and recoloring) performed via user strokes to the entire image based on similarity of image features. In most previous work, users must manually determine the importance of each image feature (e.g., color, coordinates, and textures) in accordance with their needs and target images. We focus on representation learning that automatically learns feature representations only from user strokes in a single image instead of tuning existing features manually. To this end, this paper proposes an edit propagation method using a deep neural network (DNN). Our DNN, which consists of several layers such as convolutional layers and a feature combiner, extracts stroke‐adapted visual features and spatial features, and then adjusts the importance of them. We also develop a learning algorithm for our DNN that does not suffer from the vanishing gradient problem, and hence avoids falling into undesirable locally optimal solutions. We demonstrate that edit propagation with deep features, without manual feature tuning, can achieve better results than previous work.

[1]  Zeev Farbman,et al.  Diffusion maps for edge-aware image editing , 2010, ACM Trans. Graph..

[2]  Markus Gross,et al.  Practical temporal consistency for image-based graphics applications , 2012, ACM Trans. Graph..

[3]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[4]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[5]  Harry Shum,et al.  Natural Image Colorization , 2007, Rendering Techniques.

[6]  Jun Wang,et al.  Exploring Inter-feature and Inter-class Relationships with Deep Neural Networks for Video Classification , 2014, ACM Multimedia.

[7]  Cewu Lu,et al.  Image smoothing via L0 gradient minimization , 2011, ACM Trans. Graph..

[8]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[9]  Thomas Jost,et al.  An efficient multi-algorithms sparse linear solver for GPUs , 2009, PARCO.

[10]  J. Daugman Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[11]  Nicholas D. Lane,et al.  DeepEar: robust smartphone audio sensing in unconstrained acoustic environments using deep learning , 2015, UbiComp.

[12]  Jieping Ye,et al.  A framework for interactive image color editing , 2012, The Visual Computer.

[13]  Yizhou Yu,et al.  Visual saliency based on multiscale deep features , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Tien-Tsin Wong,et al.  Manga colorization , 2006, SIGGRAPH 2006.

[15]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[16]  Shi-Min Hu,et al.  Efficient affinity-based edit propagation using K-D tree , 2009, ACM Trans. Graph..

[17]  Yan Wang,et al.  DeepContour: A deep convolutional feature learned by positive-sharing loss for contour detection , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Zeev Farbman,et al.  Interactive local adjustment of tonal values , 2006, ACM Trans. Graph..

[19]  Li Xu,et al.  A sparse control model for image and video editing , 2013, ACM Trans. Graph..

[20]  Qinping Zhao,et al.  Sparse Dictionary Learning for Edit Propagation of High-Resolution Images , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Xiaowu Chen,et al.  Manifold preserving edit propagation , 2012, ACM Trans. Graph..

[22]  Ian Reid,et al.  gSLIC: a real-time implementation of SLIC superpixel segmentation , 2011 .

[23]  Pushmeet Kohli,et al.  A perceptually motivated online benchmark for image matting , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Wencheng Wang,et al.  Real Time Edit Propagation by Efficient Sampling , 2011, Comput. Graph. Forum.

[25]  Baining Guo,et al.  Edit Propagation on Bidirectional Texture Functions , 2009, Comput. Graph. Forum.

[26]  Tien-Tsin Wong,et al.  Manga colorization , 2006, ACM Trans. Graph..

[27]  Fabio Pellacini,et al.  AppProp: all-pairs appearance-space edit propagation , 2008, SIGGRAPH 2008.

[28]  Guosheng Lin,et al.  Deep convolutional neural fields for depth estimation from a single image , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Geoffrey Zweig,et al.  Recent advances in deep learning for speech research at Microsoft , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[30]  Jason Lawrence,et al.  AppWand: editing measured materials using appearance-driven optimization , 2007, ACM Trans. Graph..

[31]  Sabine Süsstrunk,et al.  Frequency-tuned salient region detection , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Guillermo Sapiro,et al.  Fast image and video colorization using chrominance blending , 2006, IEEE Transactions on Image Processing.

[33]  Edward H. Adelson,et al.  Eurographics Symposium on Rendering 2008 Scribbleboost: Adding Classification to Edge-aware Interpolation of Local Image and Video Adjustments , 2022 .

[34]  Yizhou Yu,et al.  Automatic Photo Adjustment Using Deep Neural Networks , 2014, ACM Trans. Graph..

[35]  Dani Lischinski,et al.  Solid texture synthesis from 2D exemplars , 2007, SIGGRAPH 2007.

[36]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[37]  Shi-Min Hu,et al.  Instant Propagation of Sparse Edits on Images and Videos , 2010, Comput. Graph. Forum.

[38]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[39]  Jean Ponce,et al.  Learning a convolutional neural network for non-uniform motion blur removal , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Dani Lischinski,et al.  Colorization using optimization , 2004, ACM Trans. Graph..

[41]  Nancy Argüelles,et al.  Author ' s , 2008 .

[42]  Juhan Nam,et al.  Multimodal Deep Learning , 2011, ICML.

[43]  Dani Lischinski,et al.  Spectral Matting , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  Yuxin Peng,et al.  The application of two-level attention models in deep convolutional neural network for fine-grained image classification , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Fabio Pellacini,et al.  AppProp: all-pairs appearance-space edit propagation , 2008, ACM Trans. Graph..

[46]  Seungyong Lee,et al.  Bilateral texture filtering , 2014, ACM Trans. Graph..

[47]  Huchuan Lu,et al.  Deep networks for saliency detection via local estimation and global search , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  Quoc V. Le,et al.  On optimization methods for deep learning , 2011, ICML.

[50]  Sylvain Paris,et al.  User-assisted intrinsic images , 2009, ACM Trans. Graph..

[51]  Jiajun Wu,et al.  Deep multiple instance learning for image classification and auto-annotation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Patrick Pérez,et al.  Geodesic image and video editing , 2010, TOGS.

[53]  Yasushi Yamaguchi,et al.  Sparse pixel sampling for appearance edit propagation , 2015, The Visual Computer.

[54]  Yoshua Bengio,et al.  Greedy Layer-Wise Training of Deep Networks , 2006, NIPS.