Automated Colorization of a Grayscale Image With Seed Points Propagation

In this paper, we propose a fully automatic image colorization method for grayscale images using neural network and optimization. For a determined training set including the gray images and its corresponding color images, our method segments grayscale images into superpixels and then extracts features of particular points of interest in each superpixel. The obtained features and their RGB values are given as input for, the training colorization neural network of each pixel. To achieve a better image colorization effect in shorter running time, our method further propagates the resulting color points to neighboring pixels for improved colorization results. In the propagation of color, we present a cost function to formalize the premise that neighboring pixels should have the maximum positive similarity of intensities and colors; we then propose our solution to solving the optimization problem. At last, a guided image filter is employed to refine the colorized image. Experiments on a wide variety of images show that the proposed algorithms can achieve superior performance over the state-of-the-art algorithms.

[1]  Dani Lischinski,et al.  Colorization by example , 2005, EGSR '05.

[2]  Harry Shum,et al.  Natural Image Colorization , 2007, Rendering Techniques.

[3]  Jian Yang,et al.  From image vector to matrix: a straightforward image projection technique - IMPCA vs. PCA , 2002, Pattern Recognit..

[4]  David A. Forsyth,et al.  Learning Large-Scale Automatic Image Colorization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[5]  Krista A. Ehinger,et al.  SUN database: Large-scale scene recognition from abbey to zoo , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[6]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Santanu Chaudhury,et al.  Robust Image Colorization Using Self Attention Based Progressive Generative Adversarial Network , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[8]  J. Daugman Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[9]  Bin Sheng,et al.  Deep Colorization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[10]  Hiroshi Ishikawa,et al.  Let there be color! , 2016, ACM Trans. Graph..

[11]  Alexei A. Efros,et al.  Real-time user-guided image colorization with learned deep priors , 2017, ACM Trans. Graph..

[12]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Guillermo Sapiro,et al.  Fast image and video colorization using chrominance blending , 2006, IEEE Transactions on Image Processing.

[14]  Longin Jan Latecki,et al.  Automatic Ensemble Diffusion for 3D Shape and Image Retrieval , 2019, IEEE Transactions on Image Processing.

[15]  S. Süsstrunk,et al.  SLIC Superpixels ? , 2010 .

[16]  Stephen Lin,et al.  Intrinsic colorization , 2008, ACM Trans. Graph..

[17]  Dani Lischinski,et al.  Colorization using optimization , 2004, ACM Trans. Graph..

[18]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Chen Xu,et al.  The SUN Attribute Database: Beyond Categories for Deeper Scene Understanding , 2014, International Journal of Computer Vision.

[20]  Tao Xiang,et al.  Joint Semantic and Latent Attribute Modelling for Cross-Class Transfer Learning , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Alexei A. Efros,et al.  Colorful Image Colorization , 2016, ECCV.

[22]  Hong Li,et al.  Selective color transfer with multi-source images , 2009, Pattern Recognit. Lett..

[23]  Jian Yang,et al.  Two-dimensional PCA: a new approach to appearance-based face representation and recognition , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Noel E. O'Connor,et al.  End-to-End Conditional GAN-based Architectures for Image Colourisation , 2019, 2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP).

[25]  Djemel Ziou,et al.  Image Quality Metrics: PSNR vs. SSIM , 2010, 2010 20th International Conference on Pattern Recognition.

[26]  Tien-Tsin Wong,et al.  Manga colorization , 2006, ACM Trans. Graph..

[27]  Saman A. Zonouz,et al.  CloudID: Trustworthy cloud-based and cross-enterprise biometric identification , 2015, Expert Syst. Appl..

[28]  Jun-Cheng Chen,et al.  An adaptive edge detection based colorization algorithm and its applications , 2005, ACM Multimedia.

[29]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[30]  Tiejun Huang,et al.  Sequential Deep Trajectory Descriptor for Action Recognition With Three-Stream CNN , 2016, IEEE Transactions on Multimedia.

[31]  Vincent Lepetit,et al.  A fast local descriptor for dense matching , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[33]  Vincent Lepetit,et al.  A Fully Automated Approach to Segmentation of Irregularly Shaped Cellular Structures in EM Images , 2010, MICCAI.

[34]  Stephen Lin,et al.  Semantic colorization with internet images , 2011, ACM Trans. Graph..

[35]  Xiang Zhang,et al.  Automatic grayscale image colorization using histogram regression , 2012, Pattern Recognit. Lett..

[36]  Bernhard Schölkopf,et al.  Automatic Image Colorization Via Multimodal Predictions , 2008, ECCV.

[37]  Michael F. Cohen,et al.  Digital photography with flash and no-flash image pairs , 2004, ACM Trans. Graph..

[38]  Jian Sun,et al.  Guided Image Filtering , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[40]  Gregory Shakhnarovich,et al.  Learning Representations for Automatic Colorization , 2016, ECCV.

[41]  Deepu Rajan,et al.  Image colorization using similar images , 2012, ACM Multimedia.

[42]  Joni-Kristian Kämäräinen,et al.  Invariance properties of Gabor filter-based features-overview and applications , 2006, IEEE Transactions on Image Processing.

[43]  Kamyar Nazeri,et al.  Image Colorization with Generative Adversarial Networks , 2018, ArXiv.

[44]  Klaus Mueller,et al.  Transferring color to greyscale images , 2002, ACM Trans. Graph..