Deep binocular tone mapping

Binocular tone mapping is studied in the previous works to generate a fusible pair of LDR images in order to convey more visual content than one single LDR image. However, the existing methods are all based on monocular tone mapping operators. It greatly restricts the preservation of local details and global contrast in a binocular LDR pair. In this paper, we proposed the first binocular tone mapping operator to more effectively distribute visual content to an LDR pair, leveraging the great representability and interpretability of deep convolutional neural network. Based on the existing binocular perception models, novel loss functions are also proposed to optimize the output pairs in terms of local details, global contrast, content distribution, and binocular fusibility. Our method is validated with a qualitative and quantitative evaluation, as well as a user study. Statistics show that our method outperforms the state-of-the-art binocular tone mapping frameworks in terms of both visual quality and time performance.

[1]  W. Levelt,et al.  BINOCULAR BRIGHTNESS AVERAGING AND CONTOUR INFORMATION. , 1965, British journal of psychology.

[2]  David Zhang,et al.  FSIM: A Feature Similarity Index for Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[3]  Masahiro Yamaguchi,et al.  Comparative Evaluation of Spectral Transforms for Multispectral Image Coding in Terms of Color Degradation , 2004 .

[4]  Christophe Schlick,et al.  Quantization Techniques for Visualization of High Dynamic Range Pictures , 1995 .

[5]  Gabriel Eilertsen,et al.  HDR image reconstruction from a single exposure using deep CNNs , 2017, ACM Trans. Graph..

[6]  Tien-Tsin Wong,et al.  Binocular Tone Mapping with Improved Overall Contrast and Local Details , 2018, Comput. Graph. Forum.

[7]  Erik Reinhard,et al.  High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting , 2010 .

[8]  G. R. Engel,et al.  The autocorrelation function and binocular brightness mixing. , 1969, Vision research.

[9]  Bernhard Schölkopf,et al.  EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[10]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[11]  Greg Turk,et al.  LCIS: a boundary hierarchy for detail-preserving contrast reduction , 1999, SIGGRAPH.

[12]  D. W. Curtis,et al.  Binocular processing of brightness information: a vector-sum model. , 1978, Journal of experimental psychology. Human perception and performance.

[13]  Kurt Debattista,et al.  Advanced High Dynamic Range Imaging , 2017 .

[14]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[15]  E. Reinhard.  Photographic Tone Reproduction for Digital Images , 2002 .

[16]  Touradj Ebrahimi,et al.  Visual attention in LDR and HDR images , 2015 .

[17]  C. D. Weert,et al.  Binocular brightness combinations: Additive and nonadditive aspects , 1974 .

[18]  Jan Kautz,et al.  Local Laplacian filters: edge-aware image processing with a Laplacian pyramid , 2011, SIGGRAPH 2011.

[19]  Karol Myszkowski,et al.  Adaptive Logarithmic Mapping For Displaying High Contrast Scenes , 2003, Comput. Graph. Forum.

[20]  Vladlen Koltun,et al.  Multi-Scale Context Aggregation by Dilated Convolutions , 2015, ICLR.

[21]  Greg Ward,et al.  A Contrast-Based Scalefactor for Luminance Display , 1994, Graphics Gems.

[22]  Hugh R. Wilson,et al.  Binocular contrast, stereopsis, and rivalry: Toward a dynamical synthesis , 2017, Vision Research.

[23]  Zeev Farbman,et al.  Edge-preserving decompositions for multi-scale tone and detail manipulation , 2008, SIGGRAPH 2008.

[24]  Shengdong Zhao,et al.  ColorBless , 2015, ACM Trans. Comput. Hum. Interact..

[25]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[26]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Mark D. Fairchild,et al.  The HDR Photographic Survey , 2007, CIC.

[28]  Daniel Cohen-Or,et al.  Deep correlations for texture synthesis , 2017, TOGS.

[29]  M. Georgeson,et al.  Binocular contrast vision at and above threshold. , 2006, Journal of vision.

[30]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  S. Lippman,et al.  The Scripps Institution of Oceanography , 1959, Nature.

[32]  Murray H. Loew,et al.  Video-Level Binocular Tone-mapping Framework Based on Temporal Coherency Algorithm , 2017, 2017 IEEE Applied Imagery Pattern Recognition Workshop (AIPR).

[33]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[34]  Hermann von Helmholtz,et al.  Treatise on Physiological Optics , 1962 .

[35]  Alexei A. Efros,et al.  Fast bilateral filtering for the display of high-dynamic-range images , 2002 .

[36]  Holly E. Rushmeier,et al.  Tone reproduction for realistic images , 1993, IEEE Computer Graphics and Applications.

[37]  Hiroshi Ishikawa,et al.  Let there be color! , 2016, ACM Trans. Graph..

[38]  Frédo Durand,et al.  Fast Local Laplacian Filters , 2014, ACM Trans. Graph..

[39]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH.

[40]  Mark A. Georgeson,et al.  Interocular masking and summation indicate two stages of divisive contrast gain control , 2005 .

[41]  Yoshihiro Kanamori,et al.  Deep reverse tone mapping , 2017, ACM Trans. Graph..

[42]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[43]  Jonathan T. Barron,et al.  Burst photography for high dynamic range and low-light imaging on mobile cameras , 2016, ACM Trans. Graph..

[44]  G. Maehara,et al.  Binocular, Monocular and Dichoptic Pattern Masking , 2005 .

[45]  Hans-Peter Seidel,et al.  Beyond Tone Mapping: Enhanced Depiction of Tone Mapped HDR Images , 2006, Comput. Graph. Forum.

[46]  Scott J. Daly,et al.  Visible differences predictor: an algorithm for the assessment of image fidelity , 1992, Electronic Imaging.

[47]  Hao Li,et al.  High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Gordon E. Legge,et al.  Binocular contrast summation—II. Quadratic summation , 1984, Vision Research.

[49]  Tien-Tsin Wong,et al.  Binocular tone mapping , 2012, ACM Trans. Graph..

[50]  Jonathan T. Barron,et al.  Deep bilateral learning for real-time image enhancement , 2017, ACM Trans. Graph..

[51]  G. Legge,et al.  Binocular interactions in suprathreshold contrast perception , 1981, Perception & psychophysics.

[52]  R. Hess,et al.  Low spatial frequencies are suppressively masked across spatial scale, orientation, field position, and eye of origin. , 2004, Journal of vision.