Example-Based Image Colorization Using Locality Consistent Sparse Representation

Image colorization aims to produce a natural looking color image from a given gray-scale image, which remains a challenging problem. In this paper, we propose a novel example-based image colorization method exploiting a new locality consistent sparse representation. Given a single reference color image, our method automatically colorizes the target gray-scale image by sparse pursuit. For efficiency and robustness, our method operates at the superpixel level. We extract low-level intensity features, mid-level texture features, and high-level semantic features for each superpixel, which are then concatenated to form its descriptor. The collection of feature vectors for all the superpixels from the reference image composes the dictionary. We formulate colorization of target superpixels as a dictionary-based sparse reconstruction problem. Inspired by the observation that superpixels with similar spatial location and/or feature representation are likely to match spatially close regions from the reference image, we further introduce a locality promoting regularization term into the energy formulation, which substantially improves the matching consistency and subsequent colorization results. Target superpixels are colorized based on the chrominance information from the dominant reference superpixels. Finally, to further improve coherence while preserving sharpness, we develop a new edge-preserving filter for chrominance channels with the guidance from the target gray-scale image. To the best of our knowledge, this is the first work on sparse pursuit image colorization from single reference images. Experimental results demonstrate that our colorization method outperforms the state-of-the-art methods, both visually and quantitatively using a user study.

[1]  Jian Sun,et al.  Guided Image Filtering , 2010, ECCV.

[2]  Jun-Cheng Chen,et al.  An adaptive edge detection based colorization algorithm and its applications , 2005, ACM Multimedia.

[3]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[4]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[5]  Michael F. Cohen,et al.  Digital photography with flash and no-flash image pairs , 2004, ACM Trans. Graph..

[6]  Huchuan Lu,et al.  Saliency Detection via Graph-Based Manifold Ranking , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Aurélie Bugeau,et al.  Exemplar-based colorization in RGB color space , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[8]  Sven J. Dickinson,et al.  TurboPixels: Fast Superpixels Using Geometric Flows , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Hui Huang,et al.  Image recoloring using geodesic distance based color harmonization , 2015, Computational Visual Media.

[10]  Yu-Kun Lai,et al.  Example-based image colorization via automatic feature selection and fusion , 2017, Neurocomputing.

[11]  Yukihiko Yamashita,et al.  Self-similarity-based image colorization , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[12]  Lizhuang Ma,et al.  Gradient‐Preserving Color Transfer , 2009, Comput. Graph. Forum.

[13]  Lei Deng,et al.  Colorization Using Quaternion Algebra with Automatic Scribble Generation , 2012, MMM.

[14]  Michael F. Cohen,et al.  Fourier Analysis of the 2D Screened Poisson Equation for Gradient Domain Problems , 2008, ECCV.

[15]  Jirí Zára,et al.  Colorization of black-and-white cartoons , 2005, Image Vis. Comput..

[16]  Aurélie Bugeau,et al.  Luminance-Chrominance Model for Image Colorization , 2015, SIAM J. Imaging Sci..

[17]  Zhihua Zhang,et al.  Colorization by Matrix Completion , 2012, AAAI.

[18]  Gregory Shakhnarovich,et al.  Learning Representations for Automatic Colorization , 2016, ECCV.

[19]  Deepu Rajan,et al.  Image colorization using similar images , 2012, ACM Multimedia.

[20]  Manuel M. Oliveira,et al.  Domain transform for edge-aware image and video processing , 2011, SIGGRAPH 2011.

[21]  David A. Forsyth,et al.  Learning Large-Scale Automatic Image Colorization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[22]  Oscar C. Au,et al.  Image colorization using sparse representation , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[23]  Dongdong Nie,et al.  Optimization based grayscale image colorization , 2007, Pattern Recognit. Lett..

[24]  Xiang Zhang,et al.  Automatic grayscale image colorization using histogram regression , 2012, Pattern Recognit. Lett..

[25]  Bernhard Schölkopf,et al.  Automatic Image Colorization Via Multimodal Predictions , 2008, ECCV.

[26]  Joëlle Thollot,et al.  Automatic texture guided color transfer and colorization , 2016 .

[27]  François Pitié,et al.  Automated colour grading using colour distribution transfer , 2007, Comput. Vis. Image Underst..

[28]  Aurélie Bugeau,et al.  Variational Exemplar-Based Image Colorization , 2014, IEEE Transactions on Image Processing.

[29]  Oscar C. Au,et al.  Image colorization via color propagation and rank minimization , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[30]  Dani Lischinski,et al.  Colorization by example , 2005, EGSR '05.

[31]  Bin Sheng,et al.  Deep Colorization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[32]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[33]  Allen Y. Yang,et al.  Robust Face Recognition via Sparse Representation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Stephen Lin,et al.  Semantic colorization with internet images , 2011, ACM Trans. Graph..

[35]  Lianhong Cai,et al.  Affective Image Colorization , 2012, Journal of Computer Science and Technology.

[36]  Hideki Noda,et al.  Simple and Efficient Colorization in YCbCr Color Space , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[37]  Dani Lischinski,et al.  Colorization using optimization , 2004, ACM Trans. Graph..

[38]  Alexei A. Efros,et al.  Colorful Image Colorization , 2016, ECCV.

[39]  Erik Reinhard,et al.  Progressive histogram reshaping for creative color transfer and tone reproduction , 2010, NPAR.

[40]  Vincent Lepetit,et al.  DAISY: An Efficient Dense Descriptor Applied to Wide-Baseline Stereo , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Guillermo Sapiro,et al.  Fast image and video colorization using chrominance blending , 2006, IEEE Transactions on Image Processing.

[42]  A. Amanatiadis,et al.  Two-staged image colorization based on salient contours , 2014, 2014 IEEE International Conference on Imaging Systems and Techniques (IST) Proceedings.

[43]  Stephen Lin,et al.  Intrinsic colorization , 2008, ACM Trans. Graph..

[44]  Hiroshi Ishikawa,et al.  Let there be color! , 2016, ACM Trans. Graph..

[45]  James T. Kwok,et al.  Colorization by Patch-Based Local Low-Rank Matrix Completion , 2015, AAAI.

[46]  Lihua You,et al.  Semantic portrait color transfer with internet images , 2015, Multimedia Tools and Applications.

[47]  Li Liu,et al.  Corruptive Artifacts Suppression for Example-Based Color Transfer , 2014, IEEE Transactions on Multimedia.

[48]  O. Sorkine,et al.  Color harmonization , 2006, SIGGRAPH 2006.

[49]  Klaus Mueller,et al.  Transferring color to greyscale images , 2002, ACM Trans. Graph..