Quantum-inspired computational imaging

More to imaging than meets the eye Traditional imaging techniques involve peering down a lens and collecting as much light from the target scene as possible. That requirement can set limits on what can be seen. Altmann et al. review some of the most recent developments in the field of computational imaging, including full three-dimensional imaging of scenes that are hidden from direct view (e.g., around a corner or behind an obstacle). High-resolution imaging can be achieved with a single-pixel detector at wavelengths for which no cameras currently exist. Such advances will lead to the development of cameras that can see through fog or inside the human body. Science, this issue p. eaat2298 BACKGROUND Imaging technologies, which extend human vision capabilities, are such a natural part of our current everyday experience that we often take them for granted. However, the ability to capture images with new kinds of sensing devices that allow us to see more than what can be seen by the unaided eye has a relatively recent history. In the early 1800s, the first ever photograph was taken: an unassuming picture that required days of exposure to obtain a very grainy image. In the late 1800s, a photograph was used for the first time to see the movement of a running horse that the human eye alone could not see. In the following years, photography played a pivotal role in recording human history, ranging from influencing the creation of the first national parks in the United States all the way to documenting NASA’s Apollo 11 mission to put a man on the Moon. In the 1990s, roughly 10 billion photographs were taken per year. Facilitated by the explosion in internet usage since the 2000s, this year we will approach 2 trillion images per year—nearly 1000 images for every person on the planet. This upsurge is enabled by considerable advances in sensing and data storage and communication. At the same time, it is driving the desire for imaging technology that can further exceed the capabilities of human vision and incorporate higher-level aspects of visual processing. ADVANCES Beyond consumer products, research labs are producing new forms of imaging that look quite different from anything we were used to and, in some cases, do not resemble cameras at all. Light is typically detected at relatively high intensities, in the spectral range and with frame rates comfortable to the human eye. However, emerging technologies are now relying on sensors that can detect just one single photon, the smallest quantum out of which light is made. These detectors provide a “click,” just like a Geiger detector that clicks in the presence of radioactivity. We have now learned to use these “click” detectors to make cameras that have enhanced properties and applications. For example, videos can be created at a trillion frames per second, making a billion-fold jump in speed with respect to standard high-speed cameras. These frame rates are sufficient, for example, to freeze light in motion in the same way that previous photography techniques were able to freeze the motion of a bullet—although light travels a billion times faster than a supersonic bullet. By fusing this high temporal resolution together with single-photon sensitivity and advanced computational analysis techniques, a new generation of imaging devices is emerging, together with an unprecedented technological leap forward and new imaging applications that were previously difficult to imagine. For example, full three-dimensional (3D) images can be taken of a scene that is hidden behind a wall, the location of a person or car can be precisely tracked from behind a corner, or images can be obtained from a few photons transmitted directly through an opaque material. Inspired by quantum techniques, it is also possible to create cameras that have just one pixel or that combine information from multiple sensors, providing images with 3D and spectral information that was not otherwise possible to obtain. OUTLOOK Quantum-inspired imaging techniques combined with computational approaches and artificial intelligence are changing our perspective of what constitutes an image and what can or cannot be imaged. Steady progress is being made in the direction of building cameras that can see through fog or directly inside the human body with groundbreaking potential for self-driving cars and medical diagnosis. Other cameras are being developed that can form 3D images from information with less than one photon per pixel. Single-photon cameras have already made their way into widely sold smartphones where they are currently used for more mundane purposes such as focusing the camera lens or detecting whether the phone is being held close to one’s ear. This technology is already out of the research laboratories and is on the way to delivering fascinating imaging systems. Quantum-based imaging systems are being developed to image through opaque media (e.g., fog or human tissue) that scatter light in all directions. PHOTO: KEVIN J. MITCHELL Computational imaging combines measurement and computational methods with the aim of forming images even when the measurement conditions are weak, few in number, or highly indirect. The recent surge in quantum-inspired imaging sensors, together with a new wave of algorithms allowing on-chip, scalable and robust data processing, has induced an increase of activity with notable results in the domain of low-light flux imaging and sensing. We provide an overview of the major challenges encountered in low-illumination (e.g., ultrafast) imaging and how these problems have recently been addressed for imaging applications in extreme conditions. These methods provide examples of the future imaging solutions to be developed, for which the best results are expected to arise from an efficient codesign of the sensors and data analysis tools.

[1]  Alfred O. Hero,et al.  Nonlinear Unmixing of Hyperspectral Images: Models and Algorithms , 2013, IEEE Signal Processing Magazine.

[2]  Alfred Hero,et al.  Robust Spectral Unmixing of Sparse Multispectral Lidar Waveforms Using Gamma Markov Random Fields , 2016, IEEE Transactions on Computational Imaging.

[3]  Vivek K. Goyal,et al.  Optimal Stopping Times for Estimating Bernoulli Parameters with Applications to Active Imaging , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4]  B. Vojnovic Advanced Time‐Correlated Single Photon Counting Techniques , 2006 .

[5]  Aongus McCarthy,et al.  Spectral classification of sparse photon depth images. , 2018, Optics express.

[6]  David A. Landgrebe,et al.  Hyperspectral image data analysis , 2002, IEEE Signal Process. Mag..

[7]  Ming C. Lin,et al.  Example-guided physically based modal sound synthesis , 2013, ACM Trans. Graph..

[8]  Guang Wu,et al.  High-speed photon-counting laser ranging for broad range of distances , 2018, Scientific Reports.

[9]  Ramesh Raskar,et al.  Towards photography through realistic fog , 2018, 2018 IEEE International Conference on Computational Photography (ICCP).

[10]  Daniel J. Lum,et al.  Photon counting compressive depth mapping , 2013, Optics express.

[11]  Mingjie Sun,et al.  Adaptive foveated single-pixel imaging with dynamic supersampling , 2016, Science Advances.

[12]  W. Brockherde,et al.  CMOS Imager With 1024 SPADs and TDCs for Single-Photon Timing and 3-D Time-of-Flight , 2014, IEEE Journal of Selected Topics in Quantum Electronics.

[13]  Edoardo Charbon,et al.  Single-Photon Avalanche Diode Imagers Applied to Near-Infrared Imaging , 2014, IEEE Journal of Selected Topics in Quantum Electronics.

[14]  Antonio J. Plaza,et al.  Hyperspectral Unmixing Overview: Geometrical, Statistical, and Sparse Regression-Based Approaches , 2012, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[15]  R. Raskar,et al.  Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging , 2012, Nature Communications.

[16]  Ramesh Raskar,et al.  Lensless Imaging With Compressive Ultrafast Sensing , 2016, IEEE Transactions on Computational Imaging.

[17]  K. W. Wecht,et al.  TWO‐PHOTON EXCITATION OF FLUORESCENCE BY PICOSECOND LIGHT PULSES , 1967 .

[18]  K. Eliceiri,et al.  Non-line-of-sight imaging using a time-gated single photon avalanche diode. , 2015, Optics express.

[19]  Ramesh Raskar,et al.  Single view reflectance capture using multiplexed scattering and time-of-flight imaging , 2011, SA '11.

[20]  Edoardo Charbon,et al.  A 160×128 single-photon image sensor with on-pixel 55ps 10b time-to-digital converter , 2011, 2011 IEEE International Solid-State Circuits Conference.

[21]  Michael Wahl,et al.  Time-Correlated Single Photon Counting , 2009 .

[22]  Andrew M. Wallace,et al.  Discriminating Underwater LiDAR Target Signatures Using Sparse Multi-Spectral Depth Codes , 2016, 2016 Sensor Signal Processing for Defence (SSPD).

[23]  Jonathan Leach,et al.  Non-line-of-sight tracking of people at long range , 2017, Optics express.

[24]  G. Buller,et al.  Laser-based distance measurement using picosecond resolution time-correlated single-photon counting , 2000 .

[25]  Aongus McCarthy,et al.  Bayesian restoration of reflectivity and range profiles from subsampled single-photon multispectral Lidar data , 2017, 2017 25th European Signal Processing Conference (EUSIPCO).

[26]  Aongus McCarthy,et al.  Object Depth Profile and Reflectivity Restoration From Sparse Single-Photon Data Acquired in Underwater Environments , 2016, IEEE Transactions on Computational Imaging.

[27]  R. Raskar,et al.  Erratum: Single-photon sensitive light-in-flight imaging , 2015, Nature communications.

[28]  Aongus McCarthy,et al.  Joint range estimation and spectral classification for 3D scene reconstruction using multispectral Lidar waveforms , 2016, 2016 IEEE Statistical Signal Processing Workshop (SSP).

[29]  Jaime Martín,et al.  Tracking objects outside the line of sight using 2D intensity images , 2016, Scientific Reports.

[30]  Graham M. Gibson,et al.  Single-pixel three-dimensional imaging with time-based depth resolution , 2016, Nature Communications.

[31]  G. Buller,et al.  Kilometer-range depth imaging at 1,550 nm wavelength using an InGaAs/InP single-photon avalanche diode detector. , 2013, Optics express.

[32]  Aongus McCarthy,et al.  Joint spectral clustering and range estimation for 3D scene reconstruction using multispectral lidar waveforms , 2016, 2016 24th European Signal Processing Conference (EUSIPCO).

[33]  Alexander F. H. Goetz,et al.  Three decades of hyperspectral remote sensing of the Earth: a personal view. , 2009 .

[34]  G. Buller,et al.  Kilometer-range, high resolution depth imaging via 1560 nm wavelength single-photon detection. , 2013, Optics express.

[35]  Diego Gutierrez,et al.  Femto-photography , 2013, ACM Trans. Graph..

[36]  Vivek K Goyal,et al.  Computational multi-depth single-photon imaging. , 2016, Optics express.

[37]  Gordon Wetzstein,et al.  Towards transient imaging at interactive rates with single-photon detectors , 2018, 2018 IEEE International Conference on Computational Photography (ICCP).

[38]  Gordon Wetzstein,et al.  Single-photon 3D imaging with deep sensor fusion , 2018, ACM Trans. Graph..

[39]  Vivek K. Goyal,et al.  Photon-Efficient Computational 3-D and Reflectivity Imaging With Single-Photon Detectors , 2014, IEEE Transactions on Computational Imaging.

[40]  Steve McLaughlin,et al.  A Bayesian Approach to Denoising of Single-Photon Binary Images , 2016, IEEE Transactions on Computational Imaging.

[41]  A. Gatti,et al.  Correlated imaging, quantum and classical , 2003, quant-ph/0307187.

[42]  Martin Laurenzis,et al.  Nonline-of-sight laser gated viewing of scattered photons , 2014 .

[43]  E. R. Fossum,et al.  What to Do with Sub-Diffraction-Limit (SDL) Pixels?-A Proposal for a Gigapixel Digital Film Sensor (DFS) , 2005 .

[44]  R. Raskar,et al.  All Photons Imaging Through Volumetric Scattering , 2016, Scientific Reports.

[45]  Aongus McCarthy,et al.  Comparative study of sampling strategies for sparse photon multispectral lidar imaging: towards mosaic filter arrays , 2017 .

[46]  Shih,et al.  Optical imaging by means of two-photon quantum entanglement. , 1995, Physical review. A, Atomic, molecular, and optical physics.

[47]  Lars Sjöqvist,et al.  Scintillation index measurement using time-correlated single-photon counting laser radar , 2014 .

[48]  Naoto Yokoya,et al.  Hyperspectral Pansharpening: A Review , 2015, IEEE Geoscience and Remote Sensing Magazine.

[49]  Michael M. Kazhdan,et al.  Screened poisson surface reconstruction , 2013, TOGS.

[50]  A. Mattick,et al.  Ultrahigh speed photography of picosecond light pulses and echoes. , 1971, Applied optics.

[51]  Ximing Ren,et al.  Design and Evaluation of Multispectral LiDAR for the Recovery of Arboreal Parameters , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[52]  Matteo Perenzoni,et al.  A 32×32-pixel time-resolved single-photon image sensor with 44.64μm pitch and 19.48% fill-factor with on-chip row/frame skipping features reaching 800kHz observation rate for quantum physics applications , 2018, 2018 IEEE International Solid - State Circuits Conference - (ISSCC).

[53]  Vivek K Goyal,et al.  First-Photon Imaging , 2014, Science.

[54]  Jeffrey H. Shapiro,et al.  The physics of ghost imaging , 2012, Quantum Information Processing.

[55]  Gordon Wetzstein,et al.  Confocal non-line-of-sight imaging based on the light-cone transform , 2018, Nature.

[56]  Ting Sun,et al.  Single-pixel imaging via compressive sampling , 2008, IEEE Signal Process. Mag..

[57]  G. Buller,et al.  Underwater depth imaging using time-correlated single photon counting , 2015, Sensing Technologies + Applications.

[58]  Abderrahim Halimi,et al.  Single-photon three-dimensional imaging at up to 10 kilometers range. , 2017, Optics express.

[59]  Penelope J. Boston,et al.  Human utilization of subsurface extraterrestrial environments: Final report , 2003 .

[60]  Richard J. Hughes,et al.  Effects of propagation through atmospheric turbulence on photon statistics , 2004 .

[61]  G. Vallone,et al.  Impact of turbulence in long range quantum and classical communications. , 2012, Physical Review Letters.

[62]  Shuai Li,et al.  Lensless computational imaging through deep learning , 2017, ArXiv.

[63]  Ramesh Raskar,et al.  Object classification through scattering media with deep learning on time resolved measurement. , 2017, Optics express.

[64]  Robert W. Boyd,et al.  Imaging with a small number of photons , 2014, Nature Communications.

[65]  Graham M. Gibson,et al.  Simultaneous real-time visible and infrared video with single-pixel detectors , 2015, Scientific Reports.

[67]  Roberta E. Martin,et al.  Carnegie Airborne Observatory: in-flight fusion of hyperspectral imaging and waveform light detection and ranging for three-dimensional studies of ecosystems , 2007 .

[68]  E. Fossum,et al.  A 2.5 pJ/b Binary Image Sensor as a Pathfinder for Quanta Image Sensors , 2016, IEEE Transactions on Electron Devices.

[69]  Jon Atli Benediktsson,et al.  Advances in Spectral-Spatial Classification of Hyperspectral Images , 2013, Proceedings of the IEEE.

[70]  Jun Tanida,et al.  Object recognition through a multi-mode fiber , 2017 .

[71]  Stanley H. Chan,et al.  Optimal Threshold Design for Quanta Image Sensor , 2017, IEEE Transactions on Computational Imaging.

[72]  Ion Vornicu,et al.  Arrayable Voltage-Controlled Ring-Oscillator for Direct Time-of-Flight Image Sensors , 2017, IEEE Transactions on Circuits and Systems I: Regular Papers.

[73]  Marco N. Petrovich,et al.  Observation of laser pulse propagation in optical fibers with a SPAD camera , 2016, Scientific reports.

[74]  S. L. Thompson,et al.  Human utilization of subsurface extraterrestrial environments. , 2003, Gravitational and space biology bulletin : publication of the American Society for Gravitational and Space Biology.

[75]  Da-Wen Sun,et al.  Hyperspectral imaging for food quality analysis and control , 2010 .

[76]  Daniel Buschek,et al.  Neural network identification of people hidden from view with a single-pixel, single-photon detector , 2017, Scientific Reports.

[77]  Robert Henderson,et al.  Detection and tracking of moving objects hidden from view , 2015, Nature Photonics.

[78]  Edoardo Charbon,et al.  Nonuniformity Analysis of a 65-kpixel CMOS SPAD Imager , 2016, IEEE Transactions on Electron Devices.

[79]  Stanley H. Chan,et al.  Image Reconstruction for Quanta Image Sensors Using Deep Neural Networks , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[80]  Jian Chen,et al.  Demonstration of measuring sea fog with an SNSPD-based Lidar system , 2017, Scientific Reports.

[81]  Aongus McCarthy,et al.  Robust Bayesian Target Detection Algorithm for Depth Imaging From Sparse Single-Photon Data , 2016, IEEE Transactions on Computational Imaging.

[82]  Jeffrey H. Shapiro,et al.  Computational ghost imaging , 2008, 2009 Conference on Lasers and Electro-Optics and 2009 Conference on Quantum electronics and Laser Science Conference.

[83]  Robert Koprowski,et al.  Segmentation in dermatological hyperspectral images: dedicated methods , 2016, BioMedical Engineering OnLine.

[84]  Nicholas B. MacKinnon,et al.  Hyperspectral and Multispectral Imaging in Dermatology , 2016 .

[85]  Jon Atli Benediktsson,et al.  Recent Advances in Techniques for Hyperspectral Image Processing , 2009 .

[86]  Aongus McCarthy,et al.  Long-range depth profiling of camouflaged targets using single-photon detection , 2017 .

[87]  M. Duguay,et al.  AN ULTRAFAST LIGHT GATE , 1969 .

[88]  Vivek K. Goyal,et al.  A Few Photons Among Many: Unmixing Signal and Noise for Photon-Efficient Active Imaging , 2016, IEEE Transactions on Computational Imaging.

[89]  Vivek K. Goyal,et al.  Diffuse Imaging: Creating Optical Images With Unfocused Time-Resolved Illumination and Sensing , 2012, IEEE Signal Processing Letters.

[90]  Vivek K Goyal,et al.  Photon-efficient imaging with a single-photon camera , 2016, Nature Communications.

[91]  Colm P. O'Donnell,et al.  Hyperspectral imaging – an emerging process analytical tool for food quality and safety control , 2007 .

[92]  Aongus McCarthy,et al.  Lidar Waveform-Based Analysis of Depth Images Constructed Using Sparse Single-Photon Data , 2015, IEEE Transactions on Image Processing.

[93]  Ramesh Raskar,et al.  Estimating Motion and size of moving non-line-of-sight objects in cluttered environments , 2011, CVPR 2011.

[94]  Roderick Murray-Smith,et al.  Deep learning for real-time single-pixel video , 2018, Scientific Reports.

[95]  Vivek K. Goyal,et al.  Performance Analysis of Low-Flux Least-Squares Single-Pixel Imaging , 2016, IEEE Signal Processing Letters.