Deep Image Harmonization with Globally Guided Feature Transformation and Relation Distillation

Given a composite image, image harmonization aims to adjust the foreground illumination to be consistent with background. Previous methods have explored transforming foreground features to achieve competitive performance. In this work, we show that using global information to guide foreground feature transformation could achieve significant improvement. Besides, we propose to transfer the foreground-background relation from real images to composite images, which can provide intermediate supervision for the transformed encoder features. Additionally, considering the drawbacks of existing harmonization datasets, we also contribute a ccHarmony dataset which simulates the natural illumination variation. Extensive experiments on iHarmony4 and our contributed dataset demonstrate the superiority of our method. Our ccHarmony dataset is released at https://github.com/bcmi/Image-Harmonization-Dataset-ccHarmony.

[1]  Y. Hong,et al.  Painterly Image Harmonization in Dual Domains , 2022, AAAI.

[2]  Xintao Wang,et al.  Composite Photograph Harmonization with Complete Background Cues , 2022, ACM Multimedia.

[3]  Zhaorui Gu,et al.  Transformer for Image Harmonization and Beyond , 2022, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Yifan Liu,et al.  Semantic-guided Multi-Mask Image Harmonization , 2022, ECCV.

[5]  Rongfei Jia,et al.  DCCF: Deep Comprehensible Color Filter Learning Framework for High-Resolution Image Harmonization , 2022, ECCV.

[6]  Rynson W. H. Lau,et al.  Harmonizer: Learning to Perform White-Box Image and Video Harmonization , 2022, ECCV.

[7]  Weiyao Lin,et al.  FRIH: Fine-grained Region-aware Image Harmonization , 2022, AAAI.

[8]  Wenming Yang,et al.  SCS-Co: Self-Consistent Style Contrastive Learning for Image Harmonization , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Chunle Guo,et al.  Image Harmonization by Matching Regional References , 2022, ArXiv.

[10]  Zhaorui Gu,et al.  Image Harmonization with Transformer , 2021, IEEE International Conference on Computer Vision.

[11]  Liqing Zhang,et al.  High-Resolution Image Harmonization via Collaborative Dual Transformations , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Chi-Man Pun,et al.  Spatial-Separated Curve Rendering Network for Efficient and High-Resolution Image Harmonization , 2021, ECCV.

[13]  Zhe L. Lin,et al.  SSH: A Self-Supervised Framework for Image Harmonization , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[14]  Yuanzhen Li,et al.  Deep Image-based Illumination Harmonization , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Liqing Zhang,et al.  Making Images Real Again: A Comprehensive Survey on Deep Image Composition , 2021, ArXiv.

[16]  Haiyong Zheng,et al.  Intrinsic Image Harmonization , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Rong Xie,et al.  Region-aware Adaptive Instance Normalization for Image Harmonization , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Liqing Zhang,et al.  Deep Image Harmonization by Bridging the Reality Gap , 2021, BMVC.

[19]  Changhe Tu,et al.  Illumination Harmonization with Gray Mean Scale , 2020, CGI.

[20]  Jing Liang,et al.  Bargainnet: Background-Guided Domain Translation for Image Harmonization , 2020, 2021 IEEE International Conference on Multimedia and Expo (ICME).

[21]  Jianping Gou,et al.  Knowledge Distillation: A Survey , 2020, International Journal of Computer Vision.

[22]  Konstantin Sofiiuk,et al.  Foreground-aware Semantic Representations for Image Harmonization , 2020, 2021 IEEE Winter Conference on Applications of Computer Vision (WACV).

[23]  Anastasios Tefas,et al.  Heterogeneous Knowledge Distillation Using Information Flow Modeling , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Tero Karras,et al.  Analyzing and Improving the Image Quality of StyleGAN , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Li Niu,et al.  DoveNet: Deep Image Harmonization via Domain Verification , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Chi-Man Pun,et al.  Improving the Harmony of the Composite Image by Spatial-Separated Attention Module , 2019, IEEE Transactions on Image Processing.

[27]  Michael S. Brown,et al.  When Color Constancy Goes Wrong: Correcting Improperly White-Balanced Images , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Bor-Chun Chen,et al.  Toward Realistic Image Compositing With Adversarial Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Mao Ye,et al.  Fast Human Pose Estimation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Kun Gao,et al.  Color Correction Method for Digital Camera Based on Variable-Exponent Polynomial Regression , 2018, CSPS.

[31]  Junjie Yan,et al.  Mimicking Very Efficient Network for Object Detection , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Junmo Kim,et al.  A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Ming-Hsuan Yang,et al.  Deep Image Harmonization , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Narendra Ahuja,et al.  A Comparative Study for Single Image Blind Deblurring , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[36]  Samira Ebrahimi Kahou,et al.  FitNets: Hints for Thin Deep Nets , 2014, ICLR.

[37]  Dilip K Prasad,et al.  Illuminant estimation for color constancy: why spatial-domain methods work and the role of the color distribution. , 2014, Journal of the Optical Society of America. A, Optics, image science, and vision.

[38]  Julie Dorsey,et al.  Understanding and improving the realism of image composites , 2012, ACM Trans. Graph..

[39]  Micah K. Johnson,et al.  Multi-scale image harmonization , 2010, ACM Trans. Graph..

[40]  Andrew Blake,et al.  Bayesian color constancy revisited , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[41]  Alexei A. Efros,et al.  Using Color Compatibility for Assessing Image Realism , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[42]  E. Land.  The retinex theory of color vision. , 1977, Scientific American.

[43]  R. A. Bradley,et al.  RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS THE METHOD OF PAIRED COMPARISONS , 1952 .

[44]  Xiaoxia Wan,et al.  Estimating Polynomial Coefficients to Correct Improperly White-Balanced sRGB Images , 2021, IEEE Signal Processing Letters.

[45]  Aiguo Chen,et al.  Distilling the Knowledge of Multiscale Densely Connected Deep Networks in Mechanical Intelligent Diagnosis , 2021, Wirel. Commun. Mob. Comput..

[46]  Shengcai Liao,et al.  Exclusivity-Consistency Regularized Knowledge Distillation for Face Recognition , 2020, ECCV.

[47]  Kazuhiro Fukui,et al.  Image Harmonization with Attention-based Deep Feature Modulation , 2020, BMVC.

[48]  E. Land,et al.  Lightness and retinex theory. , 1971, Journal of the Optical Society of America.