Sun-sky model estimation from outdoor images

When a virtual object is inserted into an outdoor image, the recovery of scene illumination has a critical effect on the mix of virtual objects and actual reality. There are two main parts of the object in the outdoor scene: the sun and the sky. In order to represent the illumination conditions of these two natural illumination, this paper uses the Lalonde-Matthew outdoor illumination model to perform the sky and sun in the image. Model use seven parameters represent the illumination of the scene. So the original illumination estimation problem is transformed into a prediction problem of seven illumination parameters. For this problem, this paper proposes a new two-branch network structure, one branch is used to estimate the sun orientation, and the other branch is used to estimate the remaining six parameters. This paper also introduces convolution block attention module (CBAM) based on this structure. The introduction of this module enables the network to select the most important information for the current task target from a large number of information when extracting image features, while suppressing other useless information.

[1]  Zhong Zhou,et al.  Survey on augmented virtual environment and augmented reality , 2015 .

[2]  Alexei A. Efros,et al.  Estimating the Natural Illumination Conditions from a Single Outdoor Image , 2012, International Journal of Computer Vision.

[3]  Xiaokun Zhang,et al.  Sun orientation estimation from a single image using short-cuts in DCNN , 2019, Optics & Laser Technology.

[4]  Jean-François Lalonde,et al.  Learning to Estimate Indoor Lighting from 3D Objects , 2018, 2018 International Conference on 3D Vision (3DV).

[5]  Yannick Hold-Geoffroy,et al.  All-Weather Deep Outdoor Lighting Estimation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Jean-François Lalonde,et al.  Lighting Estimation in Outdoor Image Collections , 2014, 2014 2nd International Conference on 3D Vision.

[7]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[8]  Yannick Hold-Geoffroy,et al.  Deep Outdoor Illumination Estimation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Jitendra Malik,et al.  Shape, Illumination, and Reflectance from Shading , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Ajey Lele,et al.  Virtual reality and its military utility , 2011, Journal of Ambient Intelligence and Humanized Computing.

[11]  Taeone Kim,et al.  A practical single image based approach for estimating illumination distribution from shadows , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[12]  Gang Sun,et al.  Squeeze-and-Excitation Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[13]  Huimin Lu,et al.  Underwater Image Devignetting and Colour Correction , 2015, ICIG.

[14]  Paul Debevec Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography , 2008, SIGGRAPH Classes.

[15]  Huimin Lu,et al.  Non-uniform de-Scattering and de-Blurring of Underwater Images , 2018, Mob. Networks Appl..

[16]  Abhinav Gupta,et al.  Non-local Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[17]  Jian Shi,et al.  Learning Scene Illumination by Pairwise Photos from Rear and Front Mobile Cameras , 2018, Comput. Graph. Forum.

[18]  Qinping Zhao,et al.  A survey on virtual reality , 2009, Science in China Series F: Information Sciences.

[19]  J. Michalsky,et al.  All-weather model for sky luminance distribution—Preliminary configuration and validation , 1993 .

[20]  Nicola Capece,et al.  Engineering an advanced geo-location augmented reality framework for smart mobile devices , 2017, Journal of Ambient Intelligence and Humanized Computing.

[21]  Alexander Wilkie,et al.  Adding a Solar-Radiance Function to the Hošek-Wilkie Skylight Model , 2013, IEEE Computer Graphics and Applications.

[22]  Ersin Yumer,et al.  Learning to predict indoor illumination from a single image , 2017, ACM Trans. Graph..

[23]  Alexander Wilkie,et al.  An analytic model for full spectral sky-dome radiance , 2012, ACM Trans. Graph..

[24]  Andrew Jones,et al.  Direct HDR capture of the sun and sky , 2006, SIGGRAPH Courses.

[25]  Andrew Jones,et al.  Direct HDR capture of the sun and sky , 2004, AFRIGRAPH '04.

[26]  Krista A. Ehinger,et al.  Recognizing scene viewpoint using panoramic place representation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Sanja Fidler,et al.  Find your way by observing the sun and other semantic cues , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[28]  Han Zhang,et al.  Self-Attention Generative Adversarial Networks , 2018, ICML.

[29]  Huchuan Lu,et al.  GradNet: Gradient-Guided Network for Visual Object Tracking , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[30]  Huchuan Lu,et al.  ‘Skimming-Perusal’ Tracking: A Framework for Real-Time and Robust Long-Term Tracking , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[31]  Kevin White,et al.  Augmented reality in education and training: pedagogical approaches and illustrative case studies , 2018, J. Ambient Intell. Humaniz. Comput..

[32]  Peter Shirley,et al.  A practical analytic model for daylight , 1999, SIGGRAPH.

[33]  Huimin Lu,et al.  Turbidity Underwater Image Restoration Using Spectral Properties and Light Compensation , 2016, IEICE Trans. Inf. Syst..

[34]  Ko Nishino,et al.  Reflectance and Illumination Recovery in the Wild , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.