Blind Image Decomposition

We present and study a novel task named Blind Image Decomposition (BID), which requires separating a superimposed image into constituent underlying images in a blind setting, that is, both the source components involved in mixing as well as the mixing mechanism are unknown. For example, rain may consist of multiple components, such as rain streaks, raindrops, snow, and haze. Rainy images can be treated as an arbitrary combination of these components, some of them or all of them. How to decompose superimposed images, like rainy images, into distinct source components is a crucial step towards real-world vision systems. To facilitate research on this new task, we construct three benchmark datasets, including mixed image decomposition across multiple domains, real-scenario deraining, and joint shadow/reflection/watermark removal. Moreover, we propose a simple yet general Blind Image Decomposition Network (BIDeN) to serve as a strong baseline for future work. Experimental results demonstrate the tenability of our benchmarks and the effectiveness of BIDeN. Code and project page are available.

[1]  Shai Avidan,et al.  Non-local Image Dehazing , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  J SejnowskiTerrence,et al.  ICA Mixture Models for Unsupervised Classification of Non-Gaussian Classes and Automatic Context Switching in Blind Signal Separation , 2000 .

[3]  Mei Wang,et al.  Deep Visual Domain Adaptation: A Survey , 2018, Neurocomputing.

[4]  Hongdong Li,et al.  Beyond Monocular Deraining: Stereo Image Deraining via Semantic Understanding , 2020, ECCV.

[5]  Changshui Zhang,et al.  Blindly separating mixtures of multiple layers with spatial shifts , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Han Zhang,et al.  Self-Attention Generative Adversarial Networks , 2018, ICML.

[7]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Dongxu Li,et al.  Dual Attention-in-Attention Model for Joint Rain Streak and Raindrop Removal , 2021, IEEE Transactions on Image Processing.

[9]  Yu-Hsiang Fu,et al.  Automatic Single-Image-Based Rain Streaks Removal via Image Decomposition , 2012, IEEE Transactions on Image Processing.

[10]  Xiaochun Cao,et al.  Single Image Deraining: A Comprehensive Benchmark Analysis , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Harshad Rai,et al.  Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks , 2018 .

[13]  Vishal M. Patel,et al.  Image De-Raining Using a Conditional Generative Adversarial Network , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[14]  Cheng Lu,et al.  Entropy Minimization for Shadow Removal , 2009, International Journal of Computer Vision.

[15]  Ronen Basri,et al.  Image Segmentation by Probabilistic Bottom-Up Aggregation and Cue Integration , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Tianfan Xue,et al.  How to Train Neural Networks for Flare Removal , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[17]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[18]  Derek Hoiem,et al.  Paired Regions for Shadow Detection and Removal , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Gabriela Csurka,et al.  Deep Visual Domain Adaptation , 2020, 2020 22nd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC).

[20]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[21]  Andrzej Cichocki,et al.  Adaptive Blind Signal and Image Processing - Learning Algorithms and Applications , 2002 .

[22]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[23]  Mohinder Malhotra.  Single Image Haze Removal Using Dark Channel Prior , 2016 .

[24]  Hongyi Zhang,et al.  mixup: Beyond Empirical Risk Minimization , 2017, ICLR.

[25]  Chi-Wing Fu,et al.  Direction-Aware Spatial Context Features for Shadow Detection and Removal , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Yung-Yu Chuang,et al.  Learning to See Through Obstructions With Layered Decomposition , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Jenq-Neng Hwang,et al.  DesnowNet: Context-Aware Deep Network for Snow Removal , 2017, IEEE Transactions on Image Processing.

[28]  Adrian Galdran,et al.  Automatic Red-Channel underwater image restoration , 2015, J. Vis. Commun. Image Represent..

[29]  Zhenwei Shi,et al.  Deep Adversarial Decomposition: A Unified Framework for Separating Superimposed Images , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  William T. Freeman,et al.  A computational approach for obstruction-free photography , 2015, ACM Trans. Graph..

[31]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[32]  Yedid Hoshen,et al.  Neural separation of observed and unobserved distributions , 2018, ICML.

[33]  Changshui Zhang,et al.  Blind separation of superimposed images with unknown motions , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[35]  Rynson W. H. Lau,et al.  DeshadowNet: A Multi-context Embedding Deep Network for Shadow Removal , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Roseli A. F. Romero,et al.  Improvements on ICA mixture models for image pre-processing and segmentation , 2008, Neurocomputing.

[37]  Ren Ng,et al.  Single Image Reflection Separation with Perceptual Losses , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[38]  Yang Liu,et al.  Auto-Exposure Fusion for Single-Image Shadow Removal , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  D. Chakrabarti,et al.  A fast fixed - point algorithm for independent component analysis , 1997 .

[40]  Dawn Song,et al.  REFIT: A Unified Watermark Removal Framework For Deep Learning Systems With Limited Data , 2021, AsiaCCS.

[41]  Michal Irani,et al.  “Double-DIP”: Unsupervised Image Decomposition via Coupled Deep-Image-Priors , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Paul Newman,et al.  I Can See Clearly Now: Image Restoration via De-Raining , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[43]  Wenhan Yang,et al.  Attentive Generative Adversarial Network for Raindrop Removal from A Single Image , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[44]  Lars Petersson,et al.  Dual Contrastive Learning for Unsupervised Image-to-Image Translation , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[45]  Shree K. Nayar,et al.  Vision in bad weather , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[46]  Jinhui Tang,et al.  Single Image Dehazing via Conditional Generative Adversarial Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[47]  Luc Van Gool,et al.  Semantic Foggy Scene Understanding with Synthetic Data , 2017, International Journal of Computer Vision.

[48]  Chunxia Xiao,et al.  ARGAN: Attentive Recurrent Generative Adversarial Network for Shadow Detection and Removal , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[49]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[50]  Andrew Zisserman,et al.  A Visual Vocabulary for Flower Classification , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[51]  Narendra Ahuja,et al.  Shadow Removal Using Bilateral Filtering , 2012, IEEE Transactions on Image Processing.

[52]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Le Hui,et al.  Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[54]  Lars Petersson,et al.  Underwater Image Restoration via Contrastive Learning and a Real-world Dataset , 2021, Remote. Sens..

[55]  Han Gong,et al.  Interactive Shadow Removal and Ground Truth for Variable Scene Categories , 2014, BMVC.

[56]  Erkki Oja,et al.  Independent component analysis: algorithms and applications , 2000, Neural Networks.

[57]  James F. Blinn,et al.  A Generalization of Algebraic Surface Drawing , 1982, TOGS.

[58]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[59]  Mark D. Plumbley,et al.  Single-Channel Signal Separation and Deconvolution with Generative Adversarial Networks , 2019, IJCAI.

[60]  Lei Zhang,et al.  Joint Convolutional Analysis and Synthesis Sparse Representation for Single Image Layer Separation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[61]  Jiandong Tian,et al.  Video Desnowing and Deraining Based on Matrix Decomposition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[63]  Mohamed-Jalal Fadili,et al.  Image Decomposition and Separation Using Sparse Representations: An Overview , 2010, Proceedings of the IEEE.

[64]  Siva Karthik Mustikovela,et al.  Intrinsic Autoencoders for Joint Deferred Neural Rendering and Intrinsic Image Decomposition , 2020, 2020 International Conference on 3D Vision (3DV).

[65]  Truong Q. Nguyen,et al.  Accurate and Efficient Video De-Fencing Using Convolutional Neural Networks and Temporal Information , 2018, 2018 IEEE International Conference on Multimedia and Expo (ICME).

[66]  Ira Kemelmacher-Shlizerman,et al.  Real-Time High-Resolution Background Matting , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[67]  Dinesh Manocha,et al.  Appearance-preserving simplification , 1998, SIGGRAPH.

[68]  Raymond Y. K. Lau,et al.  Least Squares Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[69]  Robby T. Tan,et al.  Visibility in bad weather from a single image , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[70]  John Thickstun,et al.  Source Separation with Deep Generative Priors , 2020, ICML.

[71]  John E. Hopcroft,et al.  Single Image Reflection Removal Through Cascaded Refinement , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[72]  Ling Shao,et al.  Multi-Stage Progressive Image Restoration , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[73]  Jiaolong Yang,et al.  A Generic Deep Architecture for Single Image Reflection Removal and Image Smoothing (Supplementary Material) , 2017 .

[74]  Michal Irani,et al.  Co-segmentation by Composition , 2013, 2013 IEEE International Conference on Computer Vision.

[75]  Hongbin Zha,et al.  Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining , 2018, ECCV.

[76]  Dimitris Samaras,et al.  Shadow Removal via Shadow Image Decomposition , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[77]  Hongdong Li,et al.  Stereo Computation for a Single Mixture Image , 2018, ECCV.

[78]  Andrea Vedaldi,et al.  Instance Normalization: The Missing Ingredient for Fast Stylization , 2016, ArXiv.

[79]  Yang Liu,et al.  WDNet: Watermark-Decomposition Network for Visible Watermark Removal , 2020, 2021 IEEE Winter Conference on Applications of Computer Vision (WACV).

[80]  Ling-Yu Duan,et al.  Learning to Jointly Generate and Separate Reflections , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[81]  C S Asha,et al.  Auto Removal of Bright Spot from Images Captured Against Flashing Light Source , 2019, 2019 IEEE International Conference on Distributed Computing, VLSI, Electrical Circuits and Robotics (DISCOVER).

[82]  Shuicheng Yan,et al.  Deep Joint Rain Detection and Removal from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[83]  Cheng Shi,et al.  Towards Ghost-free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GAN , 2019, AAAI.

[84]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[85]  Yi Yang,et al.  Removing Raindrops and Rain Streaks in One Go , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).