GAN2C: Information Completion GAN with Dual Consistency Constraints

This paper proposes an information completion technique, GAN2C, by imposing dual consistency constraints (2C) to a closed loop encoder-decoder architecture based on the generative adversarial nets (GAN). When adopting deep neural networks as function approximators, GAN2C enables highly effective multi-modality image conversion with sparse observation in the target modes. For empirical demonstration and model evaluation, we show that trained deep neural networks in GAN2C can infer colors for grayscale images, as well as estimate rich 3D information of a scene by densely predicting the depths. The results of the experiments show that in both tasks GAN2C as a generic framework has been comparable to or advanced the state-of-the-art performance which are achieved by highly specialized systems. Code is available at https://github.com/AdalinZhang/GAN2C.

[1]  Dacheng Tao,et al.  Deep Neural Network for Structural Prediction and Lane Detection in Traffic Scene , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[2]  Chao Zhang,et al.  Transferring Colours to Grayscale Images by Locally Linear Embedding , 2008, BMVC.

[3]  Li Wang,et al.  Learning Multiviewpoint Context-Aware Representation for RGB-D Scene Classification , 2018, IEEE Signal Processing Letters.

[4]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[5]  Sergio Escalera,et al.  Organ Segmentation in Poultry Viscera Using RGB-D , 2018, Sensors.

[6]  Jung-Woo Ha,et al.  StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[7]  Léon Bottou,et al.  Wasserstein GAN , 2017, ArXiv.

[8]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[9]  Matthias Rätsch,et al.  Fast and Robust RGB-D Scene Labeling for Autonomous Driving , 2018, J. Comput..

[10]  Derek Hoiem,et al.  Indoor Segmentation and Support Inference from RGBD Images , 2012, ECCV.

[11]  Jan Kautz,et al.  Unsupervised Image-to-Image Translation Networks , 2017, NIPS.

[12]  Nizar Bouguila,et al.  A variational Bayes model for count data learning and classification , 2014, Eng. Appl. Artif. Intell..

[13]  Alexei A. Efros,et al.  Colorful Image Colorization , 2016, ECCV.

[14]  Chengqi Zhang,et al.  Learning Colours from Textures by Sparse Manifold Embedding , 2011, Australasian Conference on Artificial Intelligence.

[15]  Hua Li,et al.  Brain MR image segmentation using NAMS in pseudo-color , 2017, Computer assisted surgery.

[16]  Wei Li,et al.  Fast color transfer from multiple images , 2016, ArXiv.

[17]  Tor Lattimore,et al.  No Free Lunch versus Occam's Razor in Supervised Learning , 2011, Algorithmic Probability and Friends.

[18]  Rob Fergus,et al.  Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks , 2015, NIPS.

[19]  Yann LeCun,et al.  Deep multi-scale video prediction beyond mean square error , 2015, ICLR.

[20]  Alexei A. Efros,et al.  Real-time user-guided image colorization with learned deep priors , 2017, ACM Trans. Graph..

[21]  Mita Nasipuri,et al.  A Novel Approach for Colorization of a Grayscale Image using Soft Computing Techniques , 2017, Int. J. Multim. Data Eng. Manag..

[22]  Bernt Schiele,et al.  Generative Adversarial Text to Image Synthesis , 2016, ICML.

[23]  Jun Zhou,et al.  Manifold alignment based color transfer for multiview image stitching , 2013, 2013 IEEE International Conference on Image Processing.

[24]  Luc Van Gool,et al.  The 2017 DAVIS Challenge on Video Object Segmentation , 2017, ArXiv.

[25]  Ian D. Reid,et al.  Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Sertac Karaman,et al.  Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[27]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[28]  Dacheng Tao,et al.  A Survey on Multi-view Learning , 2013, ArXiv.

[29]  Gang Wang,et al.  Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Yong Liu,et al.  Parse geometry from a line: Monocular depth estimation with partial laser observation , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[31]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[32]  Ebrahim Abiri,et al.  Automatic colourization of grayscale images based on tensor decomposition , 2017, Multimedia Tools and Applications.

[33]  Wei Wen,et al.  Colorization of infrared images based on DWT fusion and color transfer , 2007, 2007 International Conference on Wavelet Analysis and Pattern Recognition.

[34]  Ashutosh Saxena,et al.  Learning Depth from Single Monocular Images , 2005, NIPS.

[35]  Yunhong Wang,et al.  Hierarchical Image Segmentation Ensemble for Objectness in RGB-D Images , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[36]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[37]  Habib Rostami,et al.  A New Pseudo-color Technique Based on Intensity Information Protection for Passive Sensor Imagery , 2017, ArXiv.

[38]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Nassir Navab,et al.  Deeper Depth Prediction with Fully Convolutional Residual Networks , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[40]  Klaus Mueller,et al.  Transferring color to greyscale images , 2002, ACM Trans. Graph..

[41]  Rob Fergus,et al.  Depth Map Prediction from a Single Image using a Multi-Scale Deep Network , 2014, NIPS.