PixColor: Pixel Recursive Colorization

We propose a novel approach to automatically produce multiple colorized versions of a grayscale image. Our method results from the observation that the task of automated colorization is relatively easy given a low-resolution version of the color image. We first train a conditional PixelCNN to generate a low resolution color for a given grayscale image. Then, given the generated low-resolution color image and the original grayscale image as inputs, we train a second CNN to generate a high-resolution colorization of an image. We demonstrate that our approach produces more diverse and plausible colorizations than existing methods, as judged by human raters in a "Visual Turing Test".

[1]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[2]  Zhou Wang,et al.  Multiscale structural similarity for image quality assessment , 2003, The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003.

[3]  Christoph H. Lampert,et al.  PixelCNN Models with Auxiliary Variables for Natural Image Modeling , 2017, ICML.

[4]  Stephen Lin,et al.  Semantic colorization with internet images , 2011, ACM Trans. Graph..

[5]  Hiroshi Ishikawa,et al.  Let there be color! , 2016, ACM Trans. Graph..

[6]  A. Katsaggelos,et al.  A novel visualization tool for art history and conservation: Automated colorization of black and white archival photographs of works of art , 2014 .

[7]  Mohammad Norouzi,et al.  Pixel Recursive Super Resolution , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[8]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[9]  Koray Kavukcuoglu,et al.  Pixel Recurrent Neural Networks , 2016, ICML.

[10]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[11]  Dani Lischinski,et al.  Colorization by example , 2005, EGSR '05.

[12]  Chi-Keung Tang,et al.  Local color transfer via probabilistic segmentation by expectation-maximization , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[13]  Alexei A. Efros,et al.  Colorful Image Colorization , 2016, ECCV.

[14]  Lorenzo Torresani,et al.  Multiple hypothesis colorization and its application to image compression , 2017, Comput. Vis. Image Underst..

[15]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[16]  Kevin Frans,et al.  Outline Colorization through Tandem Adversarial Networks , 2017, ArXiv.

[17]  Christoph H. Lampert,et al.  Latent Variable PixelCNNs for Natural Image Modeling , 2016 .

[18]  Xi Chen,et al.  PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications , 2017, ICLR.

[19]  Harry Shum,et al.  Natural Image Colorization , 2007, Rendering Techniques.

[20]  François Pitié,et al.  Automated colour grading using colour distribution transfer , 2007, Comput. Vis. Image Underst..

[21]  Alexei A. Efros,et al.  Real-time user-guided image colorization with learned deep priors , 2017, ACM Trans. Graph..

[22]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Guillermo Sapiro,et al.  Fast image and video colorization using chrominance blending , 2006, IEEE Transactions on Image Processing.

[24]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Gregory Shakhnarovich,et al.  Learning Representations for Automatic Colorization , 2016, ECCV.

[26]  Deepu Rajan,et al.  Image colorization using similar images , 2012, ACM Multimedia.

[27]  Yong Yu,et al.  Unsupervised Diverse Colorization via Generative Adversarial Networks , 2017, ECML/PKDD.

[28]  Dani Lischinski,et al.  Colorization using optimization , 2004, SIGGRAPH 2004.

[29]  Klaus Mueller,et al.  Transferring color to greyscale images , 2002, ACM Trans. Graph..

[30]  Bernhard Schölkopf,et al.  Automatic Image Colorization Via Multimodal Predictions , 2008, ECCV.

[31]  Thomas S. Huang,et al.  Fast Generation for Convolutional Autoregressive Models , 2017, ICLR.

[32]  Tien-Tsin Wong,et al.  Manga colorization , 2006, ACM Trans. Graph..

[33]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[34]  Alex Graves,et al.  Conditional Image Generation with PixelCNN Decoders , 2016, NIPS.

[35]  Jirí Zára,et al.  Unsupervised colorization of black-and-white cartoons , 2004, NPAR '04.

[36]  Aditya Deshpande,et al.  Learning Diverse Image Colorization , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  David A. Forsyth,et al.  Learning Large-Scale Automatic Image Colorization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[38]  Christoph H. Lampert,et al.  Probabilistic Image Colorization , 2017, BMVC.

[39]  Jun-Cheng Chen,et al.  An adaptive edge detection based colorization algorithm and its applications , 2005, ACM Multimedia.

[40]  David Pfau,et al.  Unrolled Generative Adversarial Networks , 2016, ICLR.