Neural Auto-Exposure for High-Dynamic Range Object Detection

Real-world scenes have a dynamic range of up to 280 dB that todays imaging sensors cannot directly capture. Existing live vision pipelines tackle this fundamental challenge by relying on high dynamic range (HDR) sensors that try to recover HDR images from multiple captures with different exposures. While HDR sensors substantially increase the dynamic range, they are not without disadvantages, including severe artifacts for dynamic scenes, reduced fill-factor, lower resolution, and high sensor cost. At the same time, traditional auto-exposure methods for low-dynamic range sensors have advanced as proprietary methods relying on image statistics separated from downstream vision algorithms. In this work, we revisit auto-exposure control as an alternative to HDR sensors. We propose a neural net-work for exposure selection that is trained jointly, end-to-end with an object detector and an image signal processing (ISP) pipeline. To this end, we use an HDR dataset for automotive object detection and an HDR training procedure. We validate that the proposed neural auto-exposure control, which is tailored to object detection, outperforms conventional auto-exposure methods by more than 6 points in mean average precision (mAP).

[1]  S. Iida,et al.  A 0.68e-rms Random-Noise 121dB Dynamic-Range Sub-pixel architecture CMOS Image Sensor with LED Flicker Mitigation , 2018, 2018 IEEE International Electron Devices Meeting (IEDM).

[2]  C.-C. Jay Kuo,et al.  A model-based approach to camera's auto exposure control , 2016, J. Vis. Commun. Image Represent..

[3]  Erik Reinhard,et al.  High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting , 2010 .

[4]  Thomas Bashford-Rogers,et al.  ExpandNet: A Deep Convolutional Neural Network for High Dynamic Range Expansion from Low Dynamic Range Content , 2018, Comput. Graph. Forum.

[5]  Suk-Ju Kang,et al.  Deep Recursive HDRI: Inverse Tone Mapping Using Generative Adversarial Networks , 2018, ECCV.

[6]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Richard Szeliski,et al.  High dynamic range video , 2003, ACM Trans. Graph..

[8]  M. Innocent,et al.  Pixel with nested photo diodes and 120 dB single exposure dynamic range , 2019 .

[9]  S. Kawahito,et al.  A wide dynamic range CMOS image sensor with multiple exposure-time signal outputs and 12-bit column-parallel cyclic A/D converters , 2005, IEEE Journal of Solid-State Circuits.

[10]  Minyi Guo,et al.  Personalized Exposure Control Using Adaptive Metering and Reinforcement Learning , 2019, IEEE Transactions on Visualization and Computer Graphics.

[11]  Gabriel Eilertsen,et al.  HDR image reconstruction from a single exposure using deep CNNs , 2017, ACM Trans. Graph..

[12]  Giovanni Puglisi,et al.  Image Processing for Embedded Devices , 2012 .

[13]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Erik Reinhard,et al.  Ghost Removal in High Dynamic Range Images , 2006, 2006 International Conference on Image Processing.

[15]  K. Ohno,et al.  Sub-pixel Architecture of CMOS Image Sensor Achieving over 120 dB Dynamic Range with less Motion Artifact Characteristics , 2019 .

[16]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Simon Schulz.  Using Brightness Histogram to perform Optimum Auto Exposure , 2007 .

[18]  Eli Shechtman,et al.  Robust patch-based hdr reconstruction of dynamic scenes , 2012, ACM Trans. Graph..

[19]  Jonathan T. Barron,et al.  Deep bilateral learning for real-time image enhancement , 2017, ACM Trans. Graph..

[20]  Natasha Gelfand,et al.  Multi-exposure imaging on mobile devices , 2010, ACM Multimedia.

[21]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Jonathan B. Phillips,et al.  Camera Image Quality Benchmarking , 2018 .

[23]  Quoc Kien Vuong,et al.  A New Auto Exposure and Auto White-Balance Algorithm to Detect High Dynamic Range Conditions Using CMOS Technology , 2022 .

[24]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[25]  Ce Liu,et al.  Exploring new representations and applications for motion analysis , 2009 .

[26]  Tae-Hyun Oh,et al.  Gradient-Based Camera Exposure Control for Outdoor Mobile Platforms , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[27]  Frédo Durand,et al.  Noise-optimal capture for high dynamic range photography , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[28]  Felix Heide,et al.  Hardware-in-the-Loop End-to-End Optimization of Camera Image Processing Pipelines , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Felix Heide,et al.  Hyperparameter optimization in black-box image processing using differentiable proxies , 2019, ACM Trans. Graph..

[30]  Dani Lischinski,et al.  Personalization of image enhancement , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[31]  Edward H. Adelson,et al.  Personal photo enhancement using example images , 2010, TOGS.

[32]  O. Yadid-Pecht,et al.  Wide-Dynamic-Range CMOS Image Sensors—Comparative Performance Analysis , 2009, IEEE Transactions on Electron Devices.

[33]  Marius Tico,et al.  Artifact-free High Dynamic Range imaging , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).

[34]  Sohei Manabe,et al.  A 1280 x 1080 4 . 2 μ m Split-diode Pixel HDR Sensor in 110 nm BSI CMOS Process , 2015 .

[35]  Jeff Beck,et al.  140 dB Dynamic Range Sub-electron Noise Floor Image Sensor , 2017 .

[36]  Jun Hu,et al.  HDR Deghosting: How to Deal with Saturation? , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Miguel Granados,et al.  Automatic noise modeling for ghost-free HDR reconstruction , 2013, ACM Trans. Graph..

[38]  Diego Gutierrez,et al.  Content-Aware Reverse Tone Mapping , 2016 .

[39]  EMVA Standard 1288 Standard for Characterization of Image Sensors and Cameras , 2010 .

[40]  Arnaud Darmont.  High Dynamic Range Imaging: Sensors and Architectures, Second Edition , 2019 .

[41]  Jan Kautz,et al.  Exposure Fusion , 2009, 15th Pacific Conference on Computer Graphics and Applications (PG'07).

[42]  Patrick Le Callet,et al.  High Dynamic Range Video - From Acquisition, to Display and Applications , 2016 .

[43]  Abbas El Gamal,et al.  Comparative analysis of SNR for image sensors with enhanced dynamic range , 1999, Electronic Imaging.

[44]  Ravi Ramamoorthi,et al.  Deep HDR Video from Sequences with Alternating Exposures , 2019, Comput. Graph. Forum.

[45]  Suk-Ju Kang,et al.  Deep Chain HDRI: Reconstructing a High Dynamic Range Image from a Single Low Dynamic Range Image , 2018, IEEE Access.

[46]  Panagiotis Tsakalides,et al.  Snapshot High Dynamic Range Imaging via Sparse Representations and Feature Learning , 2020, IEEE Transactions on Multimedia.

[47]  Eli Shechtman,et al.  Patch-based high dynamic range video , 2013, ACM Trans. Graph..

[48]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[49]  Shree K. Nayar,et al.  Fibonacci Exposure Bracketing for High Dynamic Range Imaging , 2013, 2013 IEEE International Conference on Computer Vision.

[50]  Jun Ohta,et al.  Smart CMOS Image Sensors and Applications , 2007 .

[51]  Frédo Durand,et al.  Differentiable programming for image processing and deep learning in halide , 2018, ACM Trans. Graph..

[52]  C.-C. Jay Kuo,et al.  Fast and robust camera's auto exposure control using convex or concave model , 2015, 2015 IEEE International Conference on Consumer Electronics (ICCE).

[53]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[54]  Wen-Chung Kao,et al.  Adaptive exposure control and real-time image fusion for surveillance systems , 2006, 2006 IEEE International Symposium on Circuits and Systems.

[55]  Steve Mann,et al.  ON BEING `UNDIGITAL' WITH DIGITAL CAMERAS: EXTENDING DYNAMIC RANGE BY COMBINING DIFFERENTLY EXPOSED PICTURES , 1995 .

[56]  Jui-Chiu Chiang,et al.  Intelligent exposure determination for high quality HDR image generation , 2013, 2013 IEEE International Conference on Image Processing.

[57]  Orly Yadid-Pecht,et al.  Wide intrascene dynamic range CMOS APS using dual sampling , 1997 .

[58]  Ming Yang,et al.  Face detection for automatic exposure control in handheld camera , 2006, Fourth IEEE International Conference on Computer Vision Systems (ICVS'06).

[59]  Yuan Cheng,et al.  Correcting over-exposure in photographs , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[60]  Shree K. Nayar,et al.  High Dynamic Range from Multiple Images: Which Exposures to Combine?∗ , 2003 .

[61]  SangHyun Park,et al.  The method of auto exposure control for low-end digital camera , 2009, 2009 11th International Conference on Advanced Communication Technology.

[62]  Sung-Jea Ko,et al.  An advanced video camera system with robust AF, AE, and AWB control , 2001, IEEE Trans. Consumer Electron..

[63]  Aykut Erdem,et al.  The State of the Art in HDR Deghosting: A Survey and Evaluation , 2015, Comput. Graph. Forum.

[64]  Roberto Manduchi,et al.  Metering for Exposure Stacks , 2012, Comput. Graph. Forum.

[65]  Ravi Ramamoorthi,et al.  Deep high dynamic range imaging of dynamic scenes , 2017, ACM Trans. Graph..

[66]  Jian Sun,et al.  Object Detection Networks on Convolutional Feature Maps , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[67]  Cheng Tan,et al.  Adaptive exposure control for image-based visual-servo systems using local gradient information. , 2020, Journal of the Optical Society of America. A, Optics, image science, and vision.