Smart Cameras

We review camera architecture in the age of artificial intelligence. Modern cameras use physical components and software to capture, compress and display image data. Over the past 5 years, deep learning solutions have become superior to traditional algorithms for each of these functions. Deep learning enables 10-100x reduction in electrical sensor power per pixel, 10x improvement in depth of field and dynamic range and 10-100x improvement in image pixel count. Deep learning enables multiframe and multiaperture solutions that fundamentally shift the goals of physical camera design. Here we review the state of the art of deep learning in camera operations and consider the impact of AI on the physical design of cameras.

[1]  Tetsuya Kuno,et al.  A new automatic exposure system for digital still cameras , 1998 .

[2]  Wen-Chung Kao,et al.  An integrated software architecture for real-time video and audio recording systems , 2005, IEEE Trans. Consumer Electron..

[3]  Nathan Hagen,et al.  Multiscale lens design. , 2009, Optics express.

[4]  Michael Elad,et al.  Fast and robust multiframe super resolution , 2004, IEEE Transactions on Image Processing.

[5]  Nasser Kehtarnavaz,et al.  A new auto-focus sharpness function for digital and smart-phone cameras , 2011, 2011 IEEE International Conference on Consumer Electronics (ICCE).

[6]  Edward H. Adelson,et al.  A multiresolution spline with application to image mosaics , 1983, TOGS.

[7]  David J. Fleet,et al.  Performance of optical flow techniques , 1994, International Journal of Computer Vision.

[8]  Wotao Yin,et al.  An Iterative Regularization Method for Total Variation-Based Image Restoration , 2005, Multiscale Model. Simul..

[9]  Thomas S. Huang,et al.  Image super-resolution as sparse representation of raw image patches , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Tomoyuki Nishita,et al.  Motion Deblurring from a Single Image using Circular Sensor Motion , 2011, Comput. Graph. Forum.

[11]  Nitin Sampat,et al.  System implications of implementing auto-exposure on consumer digital cameras , 1999, Electronic Imaging.

[12]  Richard G. Baraniuk,et al.  A new compressive imaging camera architecture using optical-domain compression , 2006, Electronic Imaging.

[13]  Shree K. Nayar,et al.  Gigapixel Computational Imaging , 2011, 2011 IEEE International Conference on Computational Photography (ICCP).

[14]  Yo-Sung Ho,et al.  Hole filling method using depth based in-painting for view synthesis in free viewpoint television and 3-D video , 2009, 2009 Picture Coding Symposium.

[15]  John von Neumann,et al.  First draft of a report on the EDVAC , 1993, IEEE Annals of the History of Computing.

[16]  C. Mead,et al.  Neuromorphic analogue VLSI. , 1995, Annual review of neuroscience.

[17]  C. Ortiz de Solórzano,et al.  Evaluation of autofocus functions in molecular cytogenetic analysis , 1997, Journal of microscopy.

[18]  W.E. Snyder,et al.  Color image processing pipeline , 2005, IEEE Signal Processing Magazine.

[19]  J. N. Mait.  A History of Imaging: Revisiting the Past to Chart the Future , 2006 .

[20]  Daniel Scharstein,et al.  Stereo vision for view synthesis , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[21]  Shree K. Nayar,et al.  Video from a single coded exposure photograph using a learned over-complete dictionary , 2011, 2011 International Conference on Computer Vision.

[22]  Vivek Agarwal,et al.  Machine learning approach to color constancy , 2007, Neural Networks.

[23]  Zhiliang Hong,et al.  Modified fast climbing search auto-focus algorithm with adaptive step size searching technique for digital camera , 2003, IEEE Trans. Consumer Electron..

[24]  Jianqin Zhou,et al.  On discrete cosine transform , 2011, ArXiv.

[25]  Marc P Christensen,et al.  Experimentally validated computational imaging with adaptive multiaperture folded architecture. , 2010, Applied optics.

[26]  Wen-Chung Kao,et al.  Adaptive exposure control and real-time image fusion for surveillance systems , 2006, 2006 IEEE International Symposium on Circuits and Systems.

[27]  Jun Tanida,et al.  Multispectral imaging using compact compound optics. , 2004, Optics express.

[28]  Michael B. Wakin,et al.  An Introduction To Compressive Sampling [A sensing/sampling paradigm that goes against the common knowledge in data acquisition] , 2008 .

[29]  Armando J. Pinho,et al.  Autonomous Configuration of Parameters in Robotic Digital Cameras , 2009, IbPRIA.

[30]  Yann LeCun,et al.  Convolutional networks and applications in vision , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[31]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[32]  Indranil Saha,et al.  journal homepage: www.elsevier.com/locate/neucom , 2022 .

[33]  Touradj Ebrahimi,et al.  The JPEG 2000 still image compression standard , 2001, IEEE Signal Process. Mag..

[34]  Keechul Jung,et al.  GPU implementation of neural networks , 2004, Pattern Recognit..

[35]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[36]  Ramesh Raskar,et al.  Computational Photography: Mastering New Techniques for Lenses, Lighting, and Sensors , 2009 .

[37]  Oscar C. Au,et al.  Recent advances in high dynamic range imaging technology , 2010, 2010 IEEE International Conference on Image Processing.

[38]  Eric R. Fossum,et al.  CMOS image sensors: electronic camera on a chip , 1995, Proceedings of International Electron Devices Meeting.

[39]  Peter Pirsch,et al.  VLSI implementations of image and video multimedia processing systems , 1998, IEEE Trans. Circuits Syst. Video Technol..

[40]  Yung-Cheng Liu,et al.  Automatic white balance of digital still camera , 1995, Proceedings of International Conference on Consumer Electronics.

[41]  P. Hanrahan,et al.  Light Field Photography with a Hand-held Plenoptic Camera , 2005 .

[42]  Irfan A. Essa,et al.  Graphcut textures: image and video synthesis using graph cuts , 2003, ACM Trans. Graph..

[43]  Frédo Durand,et al.  Image and depth from a conventional camera with a coded aperture , 2007, ACM Trans. Graph..

[44]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[45]  Evan Herbst,et al.  Occlusion Reasoning for Temporal Interpolation using Optical Flow , 2009 .

[46]  N George,et al.  Electronic imaging using a logarithmic asphere. , 2001, Optics letters.

[47]  W S McCulloch,et al.  A logical calculus of the ideas immanent in nervous activity , 1990, The Philosophy of Artificial Intelligence.

[48]  Steve Mann,et al.  ON BEING `UNDIGITAL' WITH DIGITAL CAMERAS: EXTENDING DYNAMIC RANGE BY COMBINING DIFFERENTLY EXPOSED PICTURES , 1995 .

[49]  Zhang Yan.  Design of 8 mega-pixel mobile phone camera , 2011 .

[50]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[51]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[52]  Alessandro Foi,et al.  Cross-color BM3D filtering of noisy raw data , 2009, 2009 International Workshop on Local and Non-Local Approximation in Image Processing.

[53]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[54]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, ACM Trans. Graph..

[55]  Sean M. O'Malley,et al.  A Simple, Effective System for Automated Capture of High Dynamic Range Images , 2006, Fourth IEEE International Conference on Computer Vision Systems (ICVS'06).

[56]  W T Cathey,et al.  Control of chromatic focal shift through wave-front coding. , 1998, Applied optics.

[57]  R. Szeliski,et al.  Ambient point clouds for view interpolation , 2010, ACM Trans. Graph..

[58]  Mongi A. Abidi,et al.  Evaluation of sharpness measures and search algorithms for the auto focusing of high-magnification images , 2006, SPIE Defense + Commercial Sensing.

[59]  J Ojeda-Castaneda,et al.  High focal depth by apodization and digital restoration. , 1988, Applied optics.

[60]  Qolamreza R. Razlighi,et al.  Using image entropy maximum for auto exposure , 2011, J. Electronic Imaging.

[61]  Andrew Zisserman,et al.  Automated mosaicing with super-resolution zoom , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[62]  Myunghee Cho,et al.  Fast auto-exposure algorithm based on numerical analysis , 1999, Electronic Imaging.

[63]  Jing Wang,et al.  Robust automatic white balance algorithm using gray color points in images , 2006, IEEE Transactions on Consumer Electronics.

[64]  Michael Elad,et al.  Super-Resolution Without Explicit Subpixel Motion Estimation , 2009, IEEE Transactions on Image Processing.

[65]  Steven M. Seitz,et al.  Physically-valid view synthesis by image interpolation , 1995, Proceedings IEEE Workshop on Representation of Visual Scenes (In Conjunction with ICCV'95).

[66]  Soo-Won Kim,et al.  Enhanced Autofocus Algorithm Using Robust Focus Measure and Fuzzy Reasoning , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[67]  Nasser Kehtarnavaz,et al.  Real-time face-priority auto focus for digital and cell-phone cameras , 2008, IEEE Transactions on Consumer Electronics.

[68]  L. Lucy.  An iterative technique for the rectification of observed distributions , 1974 .

[69]  Vincent Lepetit,et al.  DAISY: An Efficient Dense Descriptor Applied to Wide-Baseline Stereo , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[70]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[71]  Gary R. Bradski,et al.  ORB: An efficient alternative to SIFT or SURF , 2011, 2011 International Conference on Computer Vision.

[72]  Chang-Yeong Kim,et al.  Fast and accurate auto focusing algorithm based on two defocused images using discrete cosine transform , 2008, Electronic Imaging.

[73]  E R Dowski,et al.  Realizations of focus invariance in optical-digital systems with wave-front coding. , 1997, Applied optics.

[74]  Michal Irani,et al.  Super-resolution from a single image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[75]  Brian A. Wandell,et al.  A case for denoising before demosaicking color filter array data , 2009, 2009 Conference Record of the Forty-Third Asilomar Conference on Signals, Systems and Computers.

[76]  Shree K. Nayar,et al.  High Dynamic Range from Multiple Images: Which Exposures to Combine?∗ , 2003 .

[77]  Alexei A. Efros,et al.  Automatic photo pop-up , 2005, ACM Trans. Graph..

[78]  Mohan Shankar,et al.  Compressive video sensors using multichannel imagers. , 2010, Applied optics.

[79]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[80]  Edmund Y. Lam,et al.  Automatic White Balancing in Digital Photography , 2008 .

[81]  Takeo Kanade,et al.  Algorithms for cooperative multisensor surveillance , 2001, Proc. IEEE.

[82]  Gary J. Sullivan,et al.  Rate-distortion optimization for video compression , 1998, IEEE Signal Process. Mag..

[83]  J. P. Lewis.  Fast Normalized Cross-Correlation , 2010 .

[84]  Ramesh Raskar,et al.  Coded exposure photography: motion deblurring using fluttered shutter , 2006, SIGGRAPH '06.

[85]  Guillermo Sapiro,et al.  Non-local sparse models for image restoration , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[86]  Brian V. Funt,et al.  A comparison of computational color constancy algorithms. I: Methodology and experiments with synthesized data , 2002, IEEE Trans. Image Process..

[87]  Ajay Luthra,et al.  Overview of the H.264/AVC video coding standard , 2003, IEEE Trans. Circuits Syst. Video Technol..

[88]  Matthew A. Brown,et al.  Automatic Panoramic Image Stitching using Invariant Features , 2007, International Journal of Computer Vision.

[89]  Jitendra Malik,et al.  Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[90]  Richard Szeliski,et al.  Image mosaicing for tele-reality applications , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[91]  Gregory K. Wallace,et al.  The JPEG still picture compression standard , 1991, CACM.

[92]  Michael J. Black,et al.  Efficient Belief Propagation with Learned Higher-Order Markov Random Fields , 2006, ECCV.

[93]  JiaYi Liang,et al.  An Auto-exposure algorithm for detecting high contrast lighting conditions , 2007, 2007 7th International Conference on ASIC.

[94]  H. Sebastian Seung,et al.  Natural Image Denoising with Convolutional Networks , 2008, NIPS.

[95]  Raanan Fattal,et al.  Image and video upscaling from local self-examples , 2011, TOGS.

[96]  Masayuki Murakami,et al.  An exposure control system of video cameras based on fuzzy logic using color information , 1996, Proceedings of IEEE 5th International Fuzzy Systems.

[97]  Jean-Michel Morel,et al.  Nonlocal Image and Movie Denoising , 2008, International Journal of Computer Vision.

[98]  Peter Pirsch,et al.  VLSI architectures for video compression-a survey , 1995, Proc. IEEE.

[99]  J. A. Parker,et al.  Comparison of Interpolating Methods for Image Resampling , 1983, IEEE Transactions on Medical Imaging.

[100]  Yücel Altunbasak,et al.  Gradient based threshold free color filter array interpolation , 2010, 2010 IEEE International Conference on Image Processing.

[101]  Lei Zhang,et al.  Color demosaicking by local directional interpolation and nonlocal adaptive thresholding , 2011, J. Electronic Imaging.

[102]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[103]  E. Land.  The retinex theory of color vision. , 1977, Scientific American.

[104]  D. Psaltis,et al.  Holography in artificial neural networks , 1990, Nature.

[105]  Pierre Moulin,et al.  Frame interpolation and bidirectional prediction of video using compactly encoded optical-flow fields and label fields , 1999, IEEE Trans. Circuits Syst. Video Technol..

[106]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[107]  Michael J. Black,et al.  The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields , 1996, Comput. Vis. Image Underst..

[108]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[109]  Craig K. Rushforth,et al.  Image gathering and processing for enhanced resolution , 1984 .

[110]  E.Y. Lam,et al.  Combining gray world and retinex theory for automatic white balance in digital photography , 2005, Proceedings of the Ninth International Symposium on Consumer Electronics, 2005. (ISCE 2005)..

[111]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[112]  Andrew W. Fitzgibbon,et al.  Image-Based Rendering Using Image-Based Priors , 2005, International Journal of Computer Vision.

[113]  Steven M. Seitz,et al.  View morphing , 1996, SIGGRAPH.

[114]  Sung-Jea Ko,et al.  An advanced video camera system with robust AF, AE, and AWB control , 2001, IEEE Trans. Consumer Electron..

[115]  Giancarlo Calvagno,et al.  Color image demosaicking: An overview , 2011, Signal Process. Image Commun..

[116]  Ching-Chih Weng,et al.  A novel automatic white balance method for digital still cameras , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[117]  Lei Zhang,et al.  Color demosaicking via directional linear minimum mean square-error estimation , 2005, IEEE Transactions on Image Processing.

[118]  Oren Kapah,et al.  Demosaicking using artificial neural networks , 2000, Electronic Imaging.

[119]  Michael Mills,et al.  Panoramic overviews for navigating real-world scenes , 1993, MULTIMEDIA '93.

[120]  William H. Richardson,et al.  Bayesian-Based Iterative Method of Image Restoration , 1972 .

[121]  Peng Deng,et al.  Fast automatic white balancing method by color histogram stretching , 2011, 2011 4th International Congress on Image and Signal Processing.

[122]  Caihua Chen,et al.  Thin infrared imaging systems through multichannel sampling. , 2008, Applied optics.

[123]  Patrick J. Wolfe,et al.  Optimal exposure control for high dynamic range imaging , 2010, 2010 IEEE International Conference on Image Processing.

[124]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[125]  Jean-Michel Morel,et al.  A non-local algorithm for image denoising , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[126]  Neil Barakat,et al.  Minimal-Bracketing Sets for High-Dynamic-Range Image Capture , 2008, IEEE Transactions on Image Processing.

[127]  Jean-Michel Morel,et al.  A Review of Image Denoising Algorithms, with a New One , 2005, Multiscale Model. Simul..

[128]  Michael Elad,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries , 2006, IEEE Transactions on Image Processing.

[129]  Rey-Chue Hwang,et al.  A passive auto-focus camera control system , 2010, Appl. Soft Comput..

[130]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[131]  Uwe D. Hanebeck,et al.  Template matching using fast normalized cross correlation , 2001, SPIE Defense + Commercial Sensing.

[132]  Sung-Jea Ko,et al.  A novel training based auto-focus for mobile-phone cameras , 2011, IEEE Transactions on Consumer Electronics.

[133]  Steve Mann,et al.  Virtual bellows: constructing high quality stills from video , 1994, Proceedings of 1st International Conference on Image Processing.

[134]  Daniel L Marks,et al.  Coding for compressive focal tomography. , 2011, Applied optics.

[135]  Amine Bermak,et al.  CMOS Image Sensor with On-Chip Image Compression: A Review and Performance Analysis , 2010, J. Sensors.