Making the Invisible Visible: Highlight Substitution by Color Light Fields

Abstract In this contribution we present a new technique of high-light substitution. From a color image sequence, acquiredwith a hand-held camera, a so-calledlight field is gener-ated. Additionally, a highlight mask is calculated for eachimage of the sequence. The highlight mask is then used asa confidence map for the light field. This results in colorpixel interpolations at highlight pixels, taken from imagesin whichthese pixelswerenotover-imposedbyhighlights,resulting in better images. We demonstrate the techniqueon medical endoscopic images and evaluate the results onboth, natural and synthetic data. 1. Introduction When recording color image sequences of natural scenes,highlights due to specularreflection may considerablydis-turb the observer. This is particularly the case when med-ical images are recorded and humid tissue is subject toinspection. For endoscopic images the problem even in-creases as light source and viewing direction are almostidentical; thereby, surfaces orthogonal to the viewing di-rection are often over-imposed to such an extent, that thephysicians can only guess the tissue at that position.In this contribution we show how highlights – as wellas other image degradations– can be removed from imagesequences when a light field is created first, that is subse-quently used to enhance image quality at locations, wherethe input images show defects. The light field is a four-dimensional structure for rendering virtual color imagesfrom arbitrary positions within a certain volume.