Computational photography

Computational photography combines plentiful computing, digital sensors, modern optics, actuators, probes and smart lights to escape the limitations of traditional film cameras and enables novel imaging applications. Unbounded dynamic range, variable focus, resolution, and depth of field, hints about shape, reflectance, and lighting, and new interactive forms of photos that are partly snapshots and partly videos are just some of the new applications found in Computational Photography. The computational techniques encompass methods from modification of imaging parameters during capture to sophisticated reconstructions from indirect measurements. We provide a practical guide to topics in image capture and manipulation methods for generating compelling pictures for computer graphics and for extracting scene properties for computer vision, with several examples. Many ideas in computational photography are still relatively new to digital artists and programmers and there is no upto-date reference text. A larger problem is that a multi-disciplinary field that combines ideas from computational methods and modern digital photography involves a steep learning curve. For example, photographers are not always familiar with advanced algorithms now emerging to capture high dynamic range images, but image processing researchers face difficulty in understanding the capture and noise issues in digital cameras. These topics, however, can be easily learned without extensive background. The goal of this STAR is to present both aspects in a compact form. The new capture methods include sophisticated sensors, electromechanical actuators and on-board processing. Examples include adaptation to sensed scene depth and illumination, taking multiple pictures by varying camera parameters or actively modifying the flash illumination parameters. A class of modern reconstruction methods is also emerging. The methods can achieve a ‘photomontage’ by optimally fusing information from multiple images, improve signal to noise ratio and extract scene features such as depth edges. The STAR briefly reviews fundamental topics in digital imaging and then provides a practical guide to underlying techniques beyond image processing such as gradient domain operations, graph cuts, bilateral filters and optimizations. The participants learn about topics in image capture and manipulation methods for generating compelling pictures for computer graphics and for extracting scene properties for computer vision, with several examples. We hope to provide enough fundamentals to satisfy the technical specialist without intimidating the curious graphics researcher interested in recent advances in photography. The intended audience is photographers, digital artists, image processing programmers and vision researchers using or building applications for digital cameras or images. They will learn about camera fundamentals and powerful computational tools, along with many real world examples.

[1]  Edward H. Adelson,et al.  Single Lens Stereo with a Plenoptic Camera , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Frédo Durand,et al.  Defocus video matting , 2005, ACM Trans. Graph..

[3]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[4]  Pat Hanrahan,et al.  Conveying shape and features with image-based relighting , 2003, IEEE Visualization, 2003. VIS 2003..

[5]  Erik Reinhard,et al.  Photographic tone reproduction for digital images , 2002, ACM Trans. Graph..

[6]  Alexei A. Efros,et al.  Fast bilateral filtering for the display of high-dynamic-range images , 2002 .

[7]  Paul E. Debevec,et al.  A photometric approach to digitizing cultural artifacts , 2001, VAST '01.

[8]  Jack Tumblin,et al.  The Trilateral Filter for High Contrast Images and Meshes , 2003, Rendering Techniques.

[9]  Marc Levoy,et al.  Synthetic Aperture Focusing using a Shear-Warp Factorization of the Viewing Transform , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[10]  H. P. Robinson,et al.  Pictorial effect in photography, being hints on composition and chiaroscuro for photographers , 2022 .

[11]  Paul E. Debevec,et al.  A dual light stage , 2005, EGSR '05.

[12]  Michael F. Cohen,et al.  Photographing long scenes with multi-viewpoint panoramas , 2006, ACM Trans. Graph..

[13]  Takeo Kanade,et al.  A sorting image sensor: an example of massively parallel intensity-to-time processing for low-latency computational sensors , 1996, Proceedings of IEEE International Conference on Robotics and Automation.

[14]  S. Quegan,et al.  Understanding Synthetic Aperture Radar Images , 1998 .

[15]  Dani Lischinski,et al.  Gradient Domain High Dynamic Range Compression , 2023 .

[16]  Ania Mitros,et al.  Visual Sensor with Resolution Enhancement by Mechanical Vibrations , 2001, Auton. Robots.

[17]  Peter J. Burt,et al.  Enhanced image capture through fusion , 1993, 1993 (4th) International Conference on Computer Vision.

[18]  J. Rinehart,et al.  U . S . Patent , 2006 .

[19]  Holger Winnemöller,et al.  Light Waving: Estimating Light Positions From Photographs Alone , 2005, SIGGRAPH '05.

[20]  Kazutoshi Nakasima,et al.  Driving method for solid-state image pickup device and image pickup system , 2000 .

[21]  Yair Weiss,et al.  Deriving intrinsic images from image sequences , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[22]  Zhouchen Lin,et al.  Response to the Comments on "Fundamental Limits of Reconstruction-Based Superresolution Algorithms under Local Translation' , 2006, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Thomas F. Knight,et al.  Design of an integrated optical sensor with on-chip preprocessing , 1983 .

[24]  Andrew Gardner,et al.  Performance geometry capture for spatially varying relighting , 2005, SIGGRAPH '05.

[25]  Azriel Rosenfeld,et al.  A Fast Parallel Algorithm for Blind Estimation of Noise Variance , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, ACM Trans. Graph..

[27]  Diego A. Socolinsky Dynamic range constraints in image fusion and visualization , 2000 .

[28]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[29]  S. Kavadias,et al.  A logarithmic response CMOS image sensor with on-chip calibration , 2000, IEEE Journal of Solid-State Circuits.

[30]  J. Goodman Introduction to Fourier optics , 1969 .

[31]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[32]  Shree K. Nayar,et al.  Programmable imaging using a digital micromirror array , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[33]  Pieter Peers,et al.  Relighting with 4D incident light fields , 2003, ACM Trans. Graph..

[34]  R. D. Lockhart,et al.  The Human Figure in Motion , 1957 .

[35]  Brian A. Wandell,et al.  Rendering high dynamic range images , 2000, Electronic Imaging.

[36]  David Salesin,et al.  A Bayesian approach to digital matting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[37]  David Salesin,et al.  Panoramic video textures , 2005, ACM Trans. Graph..

[38]  Michael F. Cohen,et al.  Digital photography with flash and no-flash image pairs , 2004, ACM Trans. Graph..

[39]  P. Danielsson Euclidean distance mapping , 1980 .

[40]  VekslerOlga,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001 .

[41]  Gershon Elber Low cost illumination computation using an approximation of light wavefronts , 1994, SIGGRAPH.

[42]  Shree K. Nayar,et al.  High dynamic range imaging: spatially varying pixel exposures , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[43]  大野 茂,et al.  Polarization , 2006, A First Course in Laboratory Optics.

[44]  Richard Szeliski,et al.  Video textures , 2000, SIGGRAPH.

[45]  Shmuel Peleg,et al.  Seamless Image Stitching in the Gradient Domain , 2004, ECCV.

[46]  Richard Szeliski,et al.  High dynamic range video , 2003, ACM Trans. Graph..

[47]  Hans-Peter Seidel,et al.  Bayesian relighting , 2005, EGSR '05.

[48]  S. Ando,et al.  Time-domain correlation image sensor: First CMOS realization of demodulator pixels array , 1999 .

[49]  James F. Blinn,et al.  Blue screen matting , 1996, SIGGRAPH.

[50]  Brian A. Wandell,et al.  Multiple Capture Single Image Architecture with a CMOS Sensor , 1999 .

[51]  Steve Mann,et al.  ON BEING `UNDIGITAL' WITH DIGITAL CAMERAS: EXTENDING DYNAMIC RANGE BY COMBINING DIFFERENTLY EXPOSED PICTURES , 1995 .

[52]  Bahram Javidi,et al.  3-D computational synthetic aperture integral imaging (COMPSAII). , 2003, Optics express.

[53]  Robert Johansson,et al.  A multi-resolution 100 GOPS 4 Gpixels/s programmable CMOS image sensor for machine vision. , 2003 .

[54]  Shree K. Nayar,et al.  Assorted pixels: multi-sampled imaging with structural models , 2005, SIGGRAPH Courses.

[55]  Table-top computed lighting for practical digital photography , 2005, SIGGRAPH Courses.

[56]  Philip Dutré,et al.  Image-based lighting design , 2004 .

[57]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.

[58]  Berthold K. P. Horn SHAPE FROM SHADING: A METHOD FOR OBTAINING THE SHAPE OF A SMOOTH OPAQUE OBJECT FROM ONE VIEW , 1970 .

[59]  Tom Duff,et al.  Compositing digital images , 1984, SIGGRAPH.

[60]  Ira Kemelmacher-Shlizerman,et al.  Photometric Stereo with General, Unknown Lighting , 2006, International Journal of Computer Vision.

[61]  Walter Bender,et al.  Salient Stills: Process and Practice , 1996, IBM Syst. J..

[62]  Shree K. Nayar,et al.  Radiometric self calibration , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[63]  David Salesin,et al.  Interactive digital photomontage , 2004, ACM Trans. Graph..

[64]  Marta Braun,et al.  Picturing Time: The Work of Etienne-Jules Marey (1830-1904) , 1995 .

[65]  Shree K. Nayar,et al.  Instant dehazing of images using polarization , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[66]  William A. Barrett,et al.  Intelligent scissors for image composition , 1995, SIGGRAPH.

[67]  Greg Turk,et al.  LCIS: a boundary hierarchy for detail-preserving contrast reduction , 1999, SIGGRAPH.

[68]  W. Cathey,et al.  Extended depth of field through wave-front coding. , 1995, Applied optics.

[69]  Andrew Gardner,et al.  A lighting reproduction approach to live-action compositing , 2002, SIGGRAPH.

[70]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[71]  Julie Dorsey,et al.  Effic ient Re-rendering of Naturally Illuminated Environments , 1994 .

[72]  S. Ando,et al.  Ultrafast correlation image sensor: concept, design, and applications , 1997, Proceedings of International Solid State Sensors and Actuators Conference (Transducers '97).

[73]  Ruth Williams,et al.  Capturing the moment. , 2007, Nursing management.

[74]  Shree K. Nayar,et al.  Lensless Imaging with a Controllable Aperture , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[75]  Feng Xiao,et al.  Illuminating Illumination , 2001, CIC.

[76]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[77]  Shree K. Nayar,et al.  Motion-based motion deblurring , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[78]  Andrew Gardner,et al.  Performance relighting and reflectance transformation with time-multiplexed illumination , 2005, ACM Trans. Graph..

[79]  Jos Stam,et al.  Diffraction shaders , 1999, SIGGRAPH.

[80]  Amit K. Agrawal,et al.  Coded exposure photography: motion deblurring using fluttered shutter , 2006, ACM Trans. Graph..

[81]  Scott Mutter Surrational Images: PHOTOMONTAGES , 1992 .

[82]  Kentaro Toyama,et al.  Continuous Flash , 2003 .

[83]  Yaron Caspi,et al.  Increasing Space-Time Resolution in Video , 2002, ECCV.

[84]  Ramesh Raskar,et al.  Image fusion for context enhancement and video surrealism , 2004, NPAR '04.

[85]  Shree K. Nayar,et al.  Uncontrolled modulation imaging , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[86]  Ramesh Raskar,et al.  Why I Want a Gradient Camera , 2022 .

[87]  Cindy Grimm,et al.  Table-top computed lighting for practical digital photography , 2005, EGSR '05.

[88]  William T. Freeman,et al.  Shape-time photography , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[89]  David J. Kriegman,et al.  Image-based modeling and rendering of surfaces with arbitrary BRDFs , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[90]  Jitendra Malik,et al.  Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[91]  James Davis,et al.  Mosaics of scenes with moving objects , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[92]  David Salesin,et al.  Spatio-angular resolution tradeoffs in integral photography , 2006, EGSR '06.

[93]  B. C. Madden,et al.  Extended Intensity Range Imaging , 1993 .

[94]  Abbas El Gamal,et al.  Synthesis of high dynamic range motion blur free image from multiple captures , 2003 .

[95]  Shree K. Nayar,et al.  Removal of specularities using color and polarization , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[96]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[97]  Frédo Durand,et al.  Motion magnification , 2005, ACM Trans. Graph..

[98]  Thomas Malzbender,et al.  Polynomial texture maps , 2001, SIGGRAPH.

[99]  Brian A. Wandell,et al.  Image analysis using modulated light sources , 2001, IS&T/SPIE Electronic Imaging.

[100]  E. Adelson,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[101]  Richard Szeliski,et al.  Eliminating ghosting and exposure artifacts in image mosaics , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.