Multi-exposure photomontage with hand-held cameras

Abstract The paper studies the image fusion from multiple images taken by hand-held cameras with different exposures. Existing methods often generate unsatisfactory results, such as blurring/ghosting artifacts due to the problematic handling of camera motions, dynamic contents, and inappropriately fusion of local regions (e.g., over or under exposed). In addition, they often require a high-quality image registration, which is hard to achieve in scenarios with large depth variations and dynamic textures, and is also time-consuming. In this paper, we propose to enable a rough registration by a single homography and combine the inputs seamlessly to hide any possible misalignment. Specifically, the method first uses a Markov Random Field (MRF) energy for the labeling of all pixels, which assigns different labels to different aligned input images. During the labeling, it chooses well-exposed regions and skips moving objects at the same time. Then, the proposed method combines a Laplacian image according to the labels and constructs the fusion result by solving the Poisson equation. Furthermore, it adds some internal constraints when solving the Poisson equation for balancing and improving fusion results. We present various challenging examples, including static/dynamic, indoor/outdoor and daytime/nighttime scenes, to demonstrate the effectiveness and practicability of the proposed method.

[1]  Aykut Erdem,et al.  The State of the Art in HDR Deghosting: A Survey and Evaluation , 2015, Comput. Graph. Forum.

[2]  Charlie C. L. Wang,et al.  Gradient based image completion by solving the Poisson equation , 2007, Comput. Graph..

[3]  Zhengfang Duanmu,et al.  Deep Guided Learning for Fast Multi-Exposure Image Fusion , 2019, IEEE Transactions on Image Processing.

[4]  Tom Drummond,et al.  Machine Learning for High-Speed Corner Detection , 2006, ECCV.

[5]  Marius Tico,et al.  Artifact-free High Dynamic Range imaging , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).

[6]  Jan Kautz,et al.  Exposure Fusion , 2007, 15th Pacific Conference on Computer Graphics and Applications (PG'07).

[7]  Aykut Erdem,et al.  An Objective Deghosting Quality Metric for HDR Images , 2016, Comput. Graph. Forum.

[8]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Anna Tomaszewska,et al.  Image Registration for Multi-exposure High Dynamic Range Image Acquisition , 2007 .

[10]  Minh N. Do,et al.  Direct Photometric Alignment by Mesh Deformation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Shree K. Nayar,et al.  Radiometric self calibration , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[12]  Panajotis Agathoklis,et al.  Multi-Exposure and Multi-Focus Image Fusion in Gradient Domain , 2016, J. Circuits Syst. Comput..

[13]  Moncef Gabbouj,et al.  Joint Video Stitching and Stabilization From Moving Cameras , 2016, IEEE Transactions on Image Processing.

[14]  Zhengguo Li,et al.  Detail Preserving Multi-Scale Exposure Fusion , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[15]  Peter J. Burt,et al.  Enhanced image capture through fusion , 1993, 1993 (4th) International Conference on Computer Vision.

[16]  Kurt Debattista,et al.  Advanced High Dynamic Range Imaging: Theory and Practice , 2011 .

[17]  Zheng Liu,et al.  Objective Assessment of Multiresolution Image Fusion Algorithms for Context Enhancement in Night Vision: A Comparative Study , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  M. Hossny,et al.  Comments on 'Information measure for performance of image fusion' , 2008 .

[19]  Eli Shechtman,et al.  Robust patch-based hdr reconstruction of dynamic scenes , 2012, ACM Trans. Graph..

[20]  Eli Shechtman,et al.  Patch-based high dynamic range video , 2013, ACM Trans. Graph..

[21]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[22]  Shmuel Peleg,et al.  Seamless Image Stitching in the Gradient Domain , 2004, ECCV.

[23]  Richard Szeliski,et al.  High dynamic range video , 2003, ACM Trans. Graph..

[24]  Masahiro Okuda,et al.  Motion blur free HDR image acquisition using multiple exposures , 2008, 2008 15th IEEE International Conference on Image Processing.

[25]  Changhe Tu,et al.  An exposure fusion approach without ghost for dynamic scenes , 2013, 2013 6th International Congress on Image and Signal Processing (CISP).

[26]  Arnaud Darmont High Dynamic Range Imaging: Sensors and Architectures , 2013 .

[27]  Shree K. Nayar,et al.  Determining the Camera Response from Images: What Is Knowable? , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Shi-Min Hu,et al.  Robust background identification for dynamic video editing , 2016, ACM Trans. Graph..

[29]  Wolfgang Heidrich,et al.  HDR-VDP-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions , 2011, ACM Trans. Graph..

[30]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[31]  Chul Lee,et al.  Ghost-Free High Dynamic Range Imaging via Rank Minimization , 2014, IEEE Signal Processing Letters.

[32]  Jian Sun,et al.  Drag-and-drop pasting , 2006, SIGGRAPH 2006.

[33]  Jian Sun,et al.  Lazy snapping , 2004, SIGGRAPH 2004.

[34]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[35]  Zhengfang Duanmu,et al.  Multi-Exposure Image Fusion by Optimizing A Structural Similarity Index , 2018, IEEE Transactions on Computational Imaging.

[36]  Giuseppe Valenzise,et al.  Learning-Based Tone Mapping Operator for Efficient Image Matching , 2019, IEEE Transactions on Multimedia.

[37]  Richard Szeliski,et al.  Seamless Image Stitching of Scenes with Large Motions and Exposure Differences , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[38]  Yanning Zhang,et al.  Attention-Guided Network for Ghost-Free High Dynamic Range Imaging , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Shree K. Nayar,et al.  High dynamic range imaging: spatially varying pixel exposures , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[40]  Jian Sun,et al.  Guided Image Filtering , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  P. J. Burt,et al.  The Pyramid as a Structure for Efficient Computation , 1984 .

[42]  Tania Pouli,et al.  Towards an automatic correction of over-exposure in photographs: Application to tone-mapping , 2017, Comput. Vis. Image Underst..

[43]  Edward H. Adelson,et al.  The Laplacian Pyramid as a Compact Image Code , 1983, IEEE Trans. Commun..

[44]  Joachim Weickert,et al.  Universität Des Saarlandes Fachrichtung 6.1 – Mathematik Freehand Hdr Imaging of Moving Scenes with Simultaneous Resolution Enhancement Freehand Hdr Imaging of Moving Scenes with Simultaneous Resolution Enhancement Freehand Hdr Imaging of Moving Scenes with Simultaneous Resolution Enhancement , 2022 .

[45]  Xuelong Li,et al.  Exposure Fusion Using Boosting Laplacian Pyramid , 2014, IEEE Transactions on Cybernetics.

[46]  Miguel Granados,et al.  Automatic noise modeling for ghost-free HDR reconstruction , 2013, ACM Trans. Graph..

[47]  Takeshi Ikenaga,et al.  Ghost-free high dynamic range imaging via moving objects detection and extension , 2015, 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA).

[48]  David Salesin,et al.  Interactive digital photomontage , 2004, ACM Trans. Graph..

[49]  Lei Zhang,et al.  Multi-Exposure Fusion with CNN Features , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[50]  Ping Tan,et al.  Time slice video synthesis by robust video alignment , 2017, ACM Trans. Graph..

[51]  Jun Hu,et al.  HDR Deghosting: How to Deal with Saturation? , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[52]  Harry Shum,et al.  Lazy snapping , 2004, ACM Trans. Graph..

[53]  Wai-kuen Cham,et al.  Gradient-directed composition of multi-exposure images , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[54]  Chi-Keung Tang,et al.  Deep High Dynamic Range Imaging with Large Foreground Motions , 2017, ECCV.

[55]  Moncef Gabbouj,et al.  A Hybrid Approach for Near-Range Video Stabilization , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[56]  Steve Mann,et al.  ON BEING `UNDIGITAL' WITH DIGITAL CAMERAS: EXTENDING DYNAMIC RANGE BY COMBINING DIFFERENTLY EXPOSED PICTURES , 1995 .

[57]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[58]  Erik Reinhard,et al.  High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting , 2010 .

[59]  Dani Lischinski,et al.  Gradient Domain High Dynamic Range Compression , 2023 .

[60]  R. Venkatesh Babu,et al.  DeepFuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pairs , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[61]  Pradeep Sen,et al.  A versatile HDR video production system , 2011, ACM Trans. Graph..

[62]  Shutao Li,et al.  Fast multi-exposure image fusion with median filter and recursive filter , 2012, IEEE Transactions on Consumer Electronics.

[63]  Chuan Zhou,et al.  Gradient Based Image Completion by Solving Poisson Equation , 2005, PCM.

[64]  Shutao Li,et al.  Image Fusion With Guided Filtering , 2013, IEEE Transactions on Image Processing.

[65]  Ravi Ramamoorthi,et al.  Deep high dynamic range imaging of dynamic scenes , 2017, ACM Trans. Graph..

[66]  Yi Shen,et al.  19 – Performance evaluation of image fusion techniques , 2008 .