Single-Stage Instance Shadow Detection with Bidirectional Relation Learning

Instance shadow detection aims to find shadow instances paired with the objects that cast the shadows. The previous work adopts a two-stage framework to first predict shadow instances, object instances, and shadow-object associations from the region proposals, then leverage a post-processing to match the predictions to form the final shadow-object pairs. In this paper, we present a new single-stage fully-convolutional network architecture with a bidirectional relation learning module to directly learn the relations of shadow and object instances in an end-to-end manner. Compared with the prior work, our method actively explores the internal relationship between shadows and objects to learn a better pairing between them, thus improving the overall performance for instance shadow detection. We evaluate our method on the benchmark dataset for instance shadow detection, both quantitatively and visually. The experimental results demonstrate that our method clearly outperforms the state-of-the-art method.

[1]  Song Wang,et al.  A Multi-Task Mean Teacher for Semi-Supervised Shadow Detection , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Ross B. Girshick,et al.  Mask R-CNN , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Yi Li,et al.  Fully Convolutional Instance-Aware Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Chi-Wing Fu,et al.  Direction-Aware Spatial Context Features for Shadow Detection , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[5]  Dimitris Samaras,et al.  Noisy Label Recovery for Shadow Detection in Unfamiliar Domains , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Kaiming He,et al.  Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour , 2017, ArXiv.

[7]  Dimitris Samaras,et al.  Large-Scale Training of Shadow Detectors with Noisily-Annotated Shadow Examples , 2016, ECCV.

[8]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[9]  Jian Sun,et al.  Instance-Aware Semantic Segmentation via Multi-task Network Cascades , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Kai Chen,et al.  Hybrid Task Cascade for Instance Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Bolei Zhou,et al.  Scene Parsing through ADE20K Dataset , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Karianto Leman,et al.  Shadow optimization from structured deep edge detection , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Xiaowei Hu,et al.  Revisiting Shadow Detection: A New Benchmark Dataset for Complex World , 2019, IEEE Transactions on Image Processing.

[14]  Ronan Collobert,et al.  Learning to Refine Object Segments , 2016, ECCV.

[15]  Yung-Yu Chuang,et al.  BEDSR-Net: A Deep Shadow Removal Network From a Single Document Image , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Dimitris Samaras,et al.  Shadow Removal via Shadow Image Decomposition , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[17]  Jiejie Zhu,et al.  Learning to recognize shadows in monochromatic natural images , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[18]  Quoc V. Le,et al.  EfficientDet: Scalable and Efficient Object Detection , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Derek Hoiem,et al.  Single-image shadow detection and removal using paired regions , 2011, CVPR 2011.

[20]  Yuning Jiang,et al.  SOLO: Segmenting Objects by Locations , 2019, ECCV.

[21]  Dimitris Samaras,et al.  Shadow Detection with Conditional Generative Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[22]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[23]  Xiaowei Hu,et al.  Instance Shadow Detection , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Zhi Tian,et al.  FCOS: A Simple and Strong Anchor-Free Object Detector , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Mohammed Bennamoun,et al.  Automatic Shadow Detection and Removal from a Single Image , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Yun-Ta Tsai,et al.  Portrait shadow manipulation , 2020, ACM Trans. Graph..

[27]  Chi-Wing Fu,et al.  Bidirectional Feature Pyramid Network with Recurrent Attention Residual Modules for Shadow Detection , 2018, ECCV.

[28]  Shu Liu,et al.  Path Aggregation Network for Instance Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  Yuning Jiang,et al.  MegDet: A Large Mini-Batch Object Detector , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[30]  Cheng Shi,et al.  Towards Ghost-free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GAN , 2019, AAAI.

[31]  Alexei A. Efros,et al.  Detecting Ground Shadows in Outdoor Consumer Photographs , 2010, ECCV.

[32]  George Papandreou,et al.  MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[33]  Ronan Collobert,et al.  Learning to Segment Object Candidates , 2015, NIPS.

[34]  Xuming He,et al.  Boundary-Aware Instance Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[36]  Dimitris Samaras,et al.  Leave-One-Out Kernel Optimization for Shadow Detection , 2018, 2015 IEEE International Conference on Computer Vision (ICCV).

[37]  Le Hui,et al.  Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[38]  Chunxia Xiao,et al.  ARShadowGAN: Shadow Generative Adversarial Network for Augmented Reality in Single Light Scenes , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[40]  Ji Zhang,et al.  Large-Scale Visual Relationship Understanding , 2018, AAAI.

[41]  Hao Chen,et al.  Conditional Convolutions for Instance Segmentation , 2020, ECCV.

[42]  Jia Deng,et al.  Pixels to Graphs by Associative Embedding , 2017, NIPS.

[43]  Chi-Wing Fu,et al.  Mask-ShadowGAN: Learning to Remove Shadows From Unpaired Data , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[44]  Dimitris Samaras,et al.  A+D Net: Training a Shadow Detector with Adversarial Shadow Attenuation , 2017, ECCV.

[45]  Bolei Zhou,et al.  Semantic Understanding of Scenes Through the ADE20K Dataset , 2016, International Journal of Computer Vision.

[46]  Rynson W. H. Lau,et al.  DeshadowNet: A Multi-context Embedding Deep Network for Shadow Removal , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Ming Yang,et al.  SSAP: Single-Shot Instance Segmentation With Affinity Pyramid , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[48]  CenterMask: Real-Time Anchor-Free Instance Segmentation , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Chunxia Xiao,et al.  ARGAN: Attentive Recurrent Generative Adversarial Network for Shadow Detection and Removal , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[50]  Xiaolong Zhang,et al.  RIS-GAN: Explore Residual and Illumination with Generative Adversarial Networks for Shadow Removal , 2019, AAAI.

[51]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Dimitris Samaras,et al.  Leave-One-Out Kernel Optimization for Shadow Detection and Removal , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[53]  Hieu Le,et al.  From Shadow Segmentation to Shadow Removal , 2020, ECCV.

[54]  Touradj Ebrahimi,et al.  Cast shadow segmentation using invariant color features , 2004, Comput. Vis. Image Underst..

[55]  Minh Hoai,et al.  Large Scale Shadow Annotation and Detection Using Lazy Annotation and Stacked CNNs , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[56]  Gang Hua,et al.  What characterizes a shadow boundary under the sun and sky? , 2011, 2011 International Conference on Computer Vision.

[57]  Nikos Paragios,et al.  Illumination estimation and cast shadow detection through a higher-order graphical model , 2011, CVPR 2011.

[58]  Mohammed Bennamoun,et al.  Automatic Feature Learning for Robust Shadow Detection , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[59]  Rynson W. H. Lau,et al.  Distraction-Aware Shadow Detection , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[60]  Jiandong Tian,et al.  New spectrum ratio properties and features for shadow detection , 2016, Pattern Recognit..

[61]  Tianjia Shao,et al.  EmbedMask: Embedding Coupling for One-stage Instance Segmentation , 2019, ArXiv.

[62]  Zhuowen Tu,et al.  Aggregated Residual Transformations for Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[63]  Tao Kong,et al.  SOLOv2: Dynamic, Faster and Stronger , 2020, ArXiv.

[64]  Xinlei Chen,et al.  TensorMask: A Foundation for Dense Object Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).