CANet: A Context-Aware Network for Shadow Removal

In this paper, we propose a novel two-stage contextaware network named CANet for shadow removal, in which the contextual information from non-shadow regions is transferred to shadow regions at the embedded feature spaces. At Stage-I, we propose a contextual patch matching (CPM) module to generate a set of potential matching pairs of shadow and non-shadow patches. Combined with the potential contextual relationships between shadow and non-shadow regions, our well-designed contextual feature transfer (CFT) mechanism can transfer contextual information from non-shadow to shadow regions at different scales. With the reconstructed feature maps, we remove shadows at L and A/B channels separately. At StageII, we use an encoder-decoder to refine current results and generate the final shadow removal results. We evaluate our proposed CANet on two benchmark datasets and some real-world shadow images with complex scenes. Extensive experimental results strongly demonstrate the efficacy of our proposed CANet and exhibit superior performance to state-of-the-arts. Our source code is available at https://github.com/Zipei-Chen/CANet.

[1]  Gang Hua,et al.  What characterizes a shadow boundary under the sun and sky? , 2011, 2011 International Conference on Computer Vision.

[2]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[3]  Zhengqi Li,et al.  Learning Intrinsic Image Decomposition from Watching the World , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[4]  Rynson W. H. Lau,et al.  DeshadowNet: A Multi-context Embedding Deep Network for Shadow Removal , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Cheng Lu,et al.  On the removal of shadows from images , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Anthony Hoogs,et al.  Deep Neural Networks in Fully Connected CRF for Image Labeling with Social Network Metadata , 2018, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[7]  Gita Alaghband,et al.  Scene-LSTM: A Model for Human Trajectory Prediction , 2018, ArXiv.

[8]  Lin Chen,et al.  Efficient Shadow Removal Using Subregion Matching Illumination Transfer , 2013, Comput. Graph. Forum.

[9]  Yuxin Wang,et al.  Multi-Context And Enhanced Reconstruction Network For Single Image Super Resolution , 2020, 2020 IEEE International Conference on Multimedia and Expo (ICME).

[10]  Chi-Wing Fu,et al.  Direction-Aware Spatial Context Features for Shadow Detection and Removal , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Yi Zhang,et al.  A method of illumination effect transfer between images using color transfer and gradient fusion , 2013, 2013 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference.

[12]  Rita Cucchiara,et al.  Detecting Moving Objects, Ghosts, and Shadows in Video Streams , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  David E. Jacobs,et al.  Portrait shadow manipulation , 2020, ACM Trans. Graph..

[14]  Qing Zhang,et al.  Shadow Remover: Image Shadow Removal Based on Illumination Recovering Optimization , 2015, IEEE Transactions on Image Processing.

[15]  Rahul Sukthankar,et al.  MatchNet: Unifying feature and metric learning for patch-based matching , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Rynson W. H. Lau,et al.  Distraction-Aware Shadow Detection , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Jack Tumblin,et al.  Editing Soft Shadows in a Digital Photograph , 2007, IEEE Computer Graphics and Applications.

[18]  Chunxia Xiao,et al.  CLA‐GAN: A Context and Lightness Aware Generative Adversarial Network for Shadow Removal , 2020, Comput. Graph. Forum.

[19]  Chunxia Xiao,et al.  ARGAN: Attentive Recurrent Generative Adversarial Network for Shadow Detection and Removal , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  Song Wang,et al.  Shadow Removal by a Lightness-Guided Network With Training on Unpaired Data , 2021, IEEE Transactions on Image Processing.

[21]  Chengjiang Long,et al.  Dual Graph Convolutional Networks with Transformer and Curriculum Learning for Image Captioning , 2021, ACM Multimedia.

[22]  Le Hui,et al.  Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[23]  Gang Hua,et al.  Correlational Gaussian Processes for Cross-Domain Visual Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Bing-Yu Chen,et al.  Example‐based Multiple Local Color Transfer by Strokes , 2008, Comput. Graph. Forum.

[25]  King-Sun Fu,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence Publication Information , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Chunxia Xiao,et al.  A Multi-Task Network for Joint Specular Highlight Detection and Removal , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Chunxia Xiao,et al.  Towards High-Quality Intrinsic Images in the Wild , 2019, 2019 IEEE International Conference on Multimedia and Expo (ICME).

[29]  Yung-Yu Chuang,et al.  BEDSR-Net: A Deep Shadow Removal Network From a Single Document Image , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  V. K. Govindan,et al.  Shadow Detection and Removal from a Single Image Using LAB Color Space , 2013 .

[31]  Derek Hoiem,et al.  Single-image shadow detection and removal using paired regions , 2011, CVPR 2011.

[32]  Chunxia Xiao,et al.  RIS-GAN: Explore Residual and Illumination with Generative Adversarial Networks for Shadow Removal , 2020, AAAI.

[33]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[34]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[35]  Chunxia Xiao,et al.  A Novel Visual Representation on Text Using Diverse Conditional GAN for Visual Recognition , 2021, IEEE Transactions on Image Processing.

[36]  Rita Cucchiara,et al.  Improving shadow suppression in moving object detection with HSV color information , 2001, ITSC 2001. 2001 IEEE Intelligent Transportation Systems. Proceedings (Cat. No.01TH8585).

[37]  Mohan M. Trivedi,et al.  Moving shadow and object detection in traffic scenes , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[38]  Dimitris Samaras,et al.  Shadow Removal via Shadow Image Decomposition , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[39]  Cheng Shi,et al.  Towards Ghost-free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GAN , 2019, AAAI.

[40]  Baocai Yin,et al.  A Two-Stage Attentive Network for Single Image Super-Resolution , 2021, IEEE Transactions on Circuits and Systems for Video Technology.

[41]  Chunxia Xiao,et al.  Shadow Inpainting and Removal Using Generative Adversarial Networks with Slice Convolutions , 2019, Comput. Graph. Forum.

[42]  Chunxia Xiao,et al.  Specular Highlight Removal for Real‐world Images , 2019, Comput. Graph. Forum.

[43]  Chi-Wing Fu,et al.  Mask-ShadowGAN: Learning to Remove Shadows From Unpaired Data , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[44]  Ming Yang,et al.  Collaborative Active Visual Recognition from Crowds: A Distributed Ensemble Approach , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Cheng Lu,et al.  Entropy Minimization for Shadow Removal , 2009, International Journal of Computer Vision.

[46]  Dani Lischinski,et al.  The Shadow Meets the Mask: Pyramid‐Based Shadow Removal , 2008, Comput. Graph. Forum.