Multidimensional image retargeting

Retargeting refers to the process by which an image or video is adapted from the display device for which it was meant (target display) to another one (retarget display). The retarget display has different features from the target one such as dynamic range, discretization levels, color gamut, multi-view, and refresh rate spatial resolution. This is a very relevant topic in graphics, given the increasing number of display devices from large, high-contrast screens to small cell phones with limited dynamic range; a lot of techniques are being published in different venues, and it's hard to keep up. For most cases retargeting can be an ill-posed problem, for example in the process of displaying Low Dynamic Range (LDR) or 8-bit content on High Dynamic Range (HDR) displays. Such a problem requires the retargeting algorithm to generate new content which is missing in the input image/frame. In this course, we will present the latest solutions and techniques for retargeting images along various dimensions such as dynamic range, colors, temporal and spatial resolutions, and for the first time offer a much-needed holistic view of the field. Moreover, we are going to show how to measure and analyze the changes applied to an image or video in terms of quality using both psychophysical experiments (subjective) and computational metrics (objective). The course should be of interest to anyone involved in graphics in a broader sense, given the almost unavoidable need to retarget results to different devices -from developers interested in implementing retargeting techniques, to users that just need an overall perspective. For researchers fully engaged in developing multi-dimensional retargeting techniques, this course will serve as a solid background for future algorithms.

[1]  Charles A. Poynton,et al.  Gamma and Its Disguises : The Nonlinear Mappings of Intensity in Perception, CRTs, Film, and Video , 1993 .

[2]  Manuel Menezes de Oliveira Neto,et al.  High-quality brightness enhancement functions for real-time reverse tone mapping , 2009, The Visual Computer.

[3]  H. Ridder,et al.  Chroma variations and perceived quality of color images of natural scenes , 1997 .

[4]  Frédo Durand,et al.  A Fast Approximation of the Bilateral Filter Using a Signal Processing Approach , 2006, International Journal of Computer Vision.

[5]  Taiichiro Kurita,et al.  35.1: Moving Picture Quality Improvement for Hold‐type AM‐LCDs , 2001 .

[6]  Hans-Peter Seidel,et al.  Apparent display resolution enhancement for moving images , 2010, ACM Trans. Graph..

[7]  Takashi Okamoto,et al.  A lighting model aiming at drive simulators , 1990, SIGGRAPH.

[8]  M J Morgan,et al.  “The Analogy between Stereo Depth and Brightness”: A Reexamination , 1995, Perception.

[9]  Hans-Peter Seidel,et al.  Scalable Remote Rendering with Depth and Motion‐flow Augmented Streaming , 2011, Comput. Graph. Forum.

[10]  Andrew Adams,et al.  Fast High‐Dimensional Filtering Using the Permutohedral Lattice , 2010, Comput. Graph. Forum.

[11]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[12]  Olga Sorkine-Hornung,et al.  A comparative study of image retargeting , 2010, ACM Trans. Graph..

[13]  Ling-Yu Duan,et al.  Consumer video retargeting: context assisted spatial-temporal grid optimization , 2009, ACM Multimedia.

[14]  Przemyslaw Rokita A model for rendering high intensity lights , 1993, Comput. Graph..

[15]  Hans-Peter Seidel,et al.  Adaptive Image-space Stereo View Synthesis , 2010, VMV.

[16]  J. Goodman Introduction to Fourier optics , 1969 .

[17]  Kurt Debattista,et al.  A framework for inverse tone mapping , 2007, The Visual Computer.

[18]  M. Ronnier Luo,et al.  Calculating medium and image gamut boundaries for gamut mapping , 2000 .

[19]  J. J. Vos,et al.  Contribution of the cornea to entoptic scatter. , 1963, Journal of the Optical Society of America.

[20]  Hans-Peter Seidel,et al.  A perceptual model for disparity , 2011, ACM Trans. Graph..

[21]  Lindsay W. MacDonald Gamut Mapping in Perceptual Colour Space , 1993, Color Imaging Conference.

[22]  K. Bala,et al.  Effects of global illumination approximations on material appearance , 2010, SIGGRAPH 2010.

[23]  H.M. Wechsler,et al.  Digital image processing, 2nd ed. , 1981, Proceedings of the IEEE.

[24]  Kurt Debattista,et al.  Expanding low dynamic range videos for high dynamic range applications , 2008, SCCG.

[25]  F. Ratliff Contour and contrast. , 1972, Scientific American.

[26]  D. Burr Implications of the Craik-O'Brien illusion for brightness perception , 1987, Vision Research.

[27]  Hans-Peter Seidel,et al.  Motion-aware temporal coherence for video resizing , 2009, ACM Trans. Graph..

[28]  Aljoscha Smolic,et al.  Nonlinear disparity mapping for stereoscopic 3D , 2010, ACM Trans. Graph..

[29]  Matthias S. Keil,et al.  Gradient representations and the perception of luminosity , 2007, Vision Research.

[30]  D. Mackay,et al.  Lateral Interaction between Neural Channels sensitive to Texture Density? , 1973, Nature.

[31]  David M. Hoffman,et al.  Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. , 2008, Journal of vision.

[32]  Hans-Peter Seidel,et al.  Visible difference predicator for high dynamic range images , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[33]  Hans-Peter Seidel,et al.  A perceptual model for disparity , 2011, SIGGRAPH 2011.

[34]  Donald P. Greenberg,et al.  Physically-based glare effects for digital images , 1995, SIGGRAPH.

[35]  Ariel Shamir,et al.  Cropping Scaling Seam carving Warping Multi-operator , 2009 .

[36]  Hao Pan,et al.  LCD motion blur modeling and analysis , 2005, IEEE International Conference on Image Processing 2005.

[37]  O. Sorkine,et al.  Optimized scale-and-stretch for image resizing , 2008, SIGGRAPH 2008.

[38]  Takeshi Naemura,et al.  Glare Generation Based on Wave Optics , 2005, Comput. Graph. Forum.

[39]  Markus Gross,et al.  A system for retargeting of streaming video , 2009, SIGGRAPH 2009.

[40]  Daniel Cohen-Or,et al.  Non-homogeneous Content-driven Video-retargeting , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[41]  P. Whittle Increments and decrements: Luminance discrimination , 1986, Vision Research.

[42]  S. S. Stevens,et al.  Brightness function: effects of adaptation. , 1963, Journal of the Optical Society of America.

[43]  J. Bennett Vision and Art: The Biology of Seeing , 2003 .

[44]  K. A. Stevens,et al.  The Analogy between Stereo Depth and Brightness , 1989, Perception.

[45]  Marcus A. Magnor,et al.  Perception-motivated interpolation of image sequences , 2008, APGV.

[46]  Erik Reinhard,et al.  Do HDR displays support LDR content?: a psychophysical evaluation , 2007, ACM Trans. Graph..

[47]  D Zavagno,et al.  Some New Luminance-Gradient Effects , 1999, Perception.

[48]  Hans-Peter Seidel,et al.  Perceptually‐motivated Real‐time Temporal Upsampling of 3D Content for High‐refresh‐rate Displays , 2010, Comput. Graph. Forum.

[49]  D. Spencer,et al.  On the Stiles-Crawford effect , 1944 .

[50]  Hans-Peter Seidel,et al.  Enhancement of Bright Video Features for HDR Displays , 2008 .

[51]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[52]  Leonard McMillan,et al.  Post-rendering 3D warping , 1997, SI3D.

[53]  Hans-Peter Seidel,et al.  Brightness of the glare illusion , 2008, APGV '08.

[54]  Hans-Peter Seidel,et al.  Beyond Tone Mapping: Enhanced Depiction of Tone Mapped HDR Images , 2006, Comput. Graph. Forum.

[55]  Scott J. Daly,et al.  Visible differences predictor: an algorithm for the assessment of image fidelity , 1992, Electronic Imaging.

[56]  Kees Teunissen,et al.  Perceived sharpness in complex moving images , 1995 .

[57]  Ariel Shamir,et al.  Improved seam carving for video retargeting , 2008, SIGGRAPH 2008.

[58]  Kurt Debattista,et al.  A Psychophysical Evaluation of Inverse Tone Mapping Techniques , 2009, Comput. Graph. Forum.

[59]  D Purves,et al.  An Empirical Explanation of the Cornsweet Effect , 1999, The Journal of Neuroscience.

[60]  Hans-Peter Seidel,et al.  Spatio-temporal upsampling on the GPU , 2010, I3D '10.

[61]  Ariel Shamir,et al.  Seam Carving for Content-Aware Image Resizing , 2007, ACM Trans. Graph..

[62]  Kaleigh Smith,et al.  Contours and contrast , 2008 .

[63]  Zhou Wang,et al.  Modern Image Quality Assessment , 2006, Modern Image Quality Assessment.

[64]  G C SIMPSON,et al.  Ocular Haloes and Coronas * , 1953, The British journal of ophthalmology.

[65]  David Whitaker,et al.  Effects of luminance and external temporal noise on flicker sensitivity as a function of stimulus size at various eccentricities , 1994, Vision Research.

[66]  Jiawen Chen,et al.  Real-time edge-aware image processing with the bilateral grid , 2007, SIGGRAPH 2007.

[67]  S. Palmer Vision Science : Photons to Phenomenology , 1999 .

[68]  Michael Gleicher,et al.  Content-preserving warps for 3D video stabilization , 2009, ACM Trans. Graph..

[69]  Anthony Calabria,et al.  Perceived image contrast and observer preference , 2000 .

[70]  Hayden Landis,et al.  Production-Ready Global Illumination , 2004 .

[71]  Alan C. Bovik,et al.  A Structural Similarity Metric for Video Based on Motion Models , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[72]  Peter G. J. Barten,et al.  Contrast sensitivity of the human eye and its e ects on image quality , 1999 .

[73]  George Wolberg,et al.  Image morphing: a survey , 1998, The Visual Computer.

[74]  A. Gorea,et al.  New look at Bloch's law for contrast. , 1986, Journal of the Optical Society of America. A, Optics and image science.

[75]  Mark E. Gorzynski,et al.  CRT colorimetry. part I: Theory and practice , 1993 .

[76]  Kun Zhou,et al.  High dynamic range image hallucination , 2007, SIGGRAPH '07.

[77]  Francesco Banterle,et al.  Inverse tone mapping , 2006, GRAPHITE '06.

[78]  C. Tyler Spatial organization of binocular disparity sensitivity , 1975, Vision Research.

[79]  Hans-Peter Seidel,et al.  Predicting visible differences in high dynamic range images: model and its calibration , 2005, IS&T/SPIE Electronic Imaging.

[80]  Hans-Peter Seidel,et al.  Apparent resolution enhancement for animations , 2011, SCC.

[81]  Joris E Coppens,et al.  The ciliary corona: physical model and simulation of the fine needles radiating from point light sources. , 2005, Investigative ophthalmology & visual science.

[82]  Leo Jan Velthoven,et al.  Motion blur reduction for liquid crystal displays: motion-compensated inverse filtering , 2004, IS&T/SPIE Electronic Imaging.

[83]  Karol Myszkowski,et al.  3D unsharp masking for scene coherent enhancement , 2008, SIGGRAPH 2008.

[84]  Albert J. Ahumada,et al.  Computational image quality metrics: A review , 1993 .

[85]  George Drettakis,et al.  Interactive Rendering using the Render Cache , 1999, Rendering Techniques.

[86]  Jan Morovic,et al.  To develop a universal gamut mapping algorithm , 1998 .

[87]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[88]  Nicolas Bonnier,et al.  Spatial and Color Adaptive Gamut Mapping: A Mathematical Framework and Two New Algorithms , 2007, CIC.

[89]  M. Levoy,et al.  Gaussian KD-trees for fast high-dimensional filtering , 2009, SIGGRAPH 2009.

[90]  Diego Gutierrez,et al.  Evaluation of reverse tone mapping through varying exposure conditions , 2009, SIGGRAPH 2009.

[91]  Mark D. Fairchild,et al.  General-Purpose Gamut-Mapping Algorithms: Evaluation of Contrast-Preserving Rescaling Functions for Color Gamut Mapping , 1999, CIC.

[92]  W N Charman,et al.  A simple parametric model of the human ocular modulation transfer function , 1991, Ophthalmic & physiological optics : the journal of the British College of Ophthalmic Opticians.

[93]  Xiao-Fan Feng LCD motion-blur analysis, perception, and reduction using synchronized backlight flashing , 2006, Electronic Imaging.

[94]  Hao Pan,et al.  Comparisons of motion-blur assessment strategies for newly emergent LCD and backlight driving technologies , 2008 .

[95]  G. Caputo,et al.  The Glare Effect and the Perception of Luminosity , 2001, Perception.

[96]  Hans-Peter Seidel,et al.  Contrast Restoration by Adaptive Countershading , 2007, Comput. Graph. Forum.

[97]  Tong-Yee Lee,et al.  Motion-based video retargeting with optimized crop-and-warp , 2010, SIGGRAPH 2010.

[98]  Craig Gotsman,et al.  Energy‐Based Image Deformation , 2009, Comput. Graph. Forum.

[99]  Ian P. Howard,et al.  A Craik-O'Brien-Cornsweet illusion for visual depth , 1978, Vision Research.

[100]  Hans-Peter Seidel,et al.  Perception-motivated high dynamic range video encoding , 2004, SIGGRAPH 2004.

[101]  Jason Lawrence,et al.  Accelerating real-time shading with reverse reprojection caching , 2007, GH '07.

[102]  Dietmar Wüller,et al.  The usage of digital cameras as luminance meters , 2007, Electronic Imaging.

[103]  Laurence Meylan,et al.  The Reproduction of Specular Highlights on High Dynamic Range Displays , 2006, CIC.

[104]  R P Dooley,et al.  Measurements of edge-induced visual contrast and a spatial-frequency interaction of the Cornsweet illusion. , 1977, Journal of the Optical Society of America.

[105]  Yael Pritch,et al.  Shift-map image editing , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[106]  M D'Zmura,et al.  Mechanisms of color constancy. , 1986, Journal of the Optical Society of America. A, Optics and image science.

[107]  Hans-Peter Seidel,et al.  Dynamic range independent image quality assessment , 2008, ACM Trans. Graph..

[108]  Hans-Peter Seidel,et al.  Temporal Glare: Real‐Time Dynamic Simulation of the Scattering in the Human Eye , 2009, Comput. Graph. Forum.

[109]  A. Oppenheim,et al.  Nonlinear filtering of multiplied and convolved signals , 1968 .

[110]  Miranda Mn The eye as an optical instrument , 1978 .

[111]  László Neumann,et al.  Accurate display gamma functions based on human observation , 2007 .

[112]  Karol Myszkowski,et al.  Apparent Greyscale: A Simple and Fast Conversion to Perceptually Accurate Images and Video , 2008, Comput. Graph. Forum.

[113]  David M. Hoffman,et al.  The zone of comfort: Predicting visual discomfort with stereo displays. , 2011, Journal of vision.

[114]  John J. McCann Do humans discount the illuminant? , 2005, IS&T/SPIE Electronic Imaging.

[115]  Wolfgang Heidrich,et al.  Color correction for tone mapping , 2009, Comput. Graph. Forum.

[116]  Sung-Soo Kim,et al.  Nonlinearity compensated smooth frame insertion for motion-blur reduction in LCD , 2005, 2005 IEEE 7th Workshop on Multimedia Signal Processing.

[117]  F. Kingdom,et al.  Border effects on brightness: a review of findings, models and issues. , 1988, Spatial vision.

[118]  Lei Yang,et al.  A Survey on Temporal Coherence Methods in Real-Time Rendering , 2011, Eurographics.

[119]  A. Hendrickson,et al.  Human photoreceptor topography , 1990, The Journal of comparative neurology.

[120]  B. Rogers,et al.  Anisotropies in the perception of three-dimensional surfaces. , 1983, Science.

[121]  Bruce Walter,et al.  Visual equivalence: towards a new standard for image fidelity , 2007, SIGGRAPH 2007.

[122]  Hans-Peter Seidel,et al.  Physically-based real-time lens flare rendering , 2011, ACM Trans. Graph..

[123]  Oliver Deussen,et al.  Image enhancement by unsharp masking the depth buffer , 2006, SIGGRAPH 2006.

[124]  Wolfgang Heidrich,et al.  Ldr2Hdr: on-the-fly reverse tone mapping of legacy video and photographs , 2007, SIGGRAPH 2007.

[125]  Takeshi Naemura,et al.  Glare simulation and its application to evaluation of bright lights with spectral power distribution , 2005, SIGGRAPH '05.

[126]  Laurence Meylan,et al.  Tone mapping for high dynamic range displays , 2007, Electronic Imaging.

[127]  Mark D. Fairchild,et al.  Perceived Image Contrast and Observer Preference I. The Effects of Lightness, Chroma, and Sharpness Manipulations on Contrast Perception , 2003, Journal of Imaging Science and Technology.

[128]  Hans-Peter Seidel,et al.  Apparent display resolution enhancement for moving images , 2010, SIGGRAPH 2010.

[129]  Weisi Lin,et al.  Perceptual impact of edge sharpness in images , 2006 .

[130]  Jung Hong Chuang Level of Detail for 3D Graphics , 2002 .

[131]  Jason Lawrence,et al.  Automated reprojection-based pixel shader optimization , 2008, SIGGRAPH Asia '08.

[132]  P. Whittle,et al.  Contrast discrimination at high contrasts reveals the influence of local light adaptation on contrast processing , 1996, Vision Research.

[133]  Kenneth Chiu,et al.  Spatially Nonuniform Scaling Functions for High Contrast Images , 1993 .

[134]  James E. Cutting,et al.  Chapter 3 – Perceiving Layout and Knowing Distances: The Integration, Relative Potency, and Contextual Use of Different Information about Depth* , 1995 .

[135]  Hans-Peter Seidel,et al.  Analysis of Reproducing Real‐World Appearance on Displays of Varying Dynamic Range , 2006, Comput. Graph. Forum.

[136]  Wojciech Matusik,et al.  Moving gradients: a path-based method for plausible image interpolation , 2009, ACM Trans. Graph..

[137]  Hans-Peter Seidel,et al.  Video quality assessment for computer graphics applications , 2010, ACM Trans. Graph..

[138]  Mark F. Bradshaw,et al.  Sensitivity to horizontal and vertical corrugations defined by binocular disparity , 1999, Vision Research.