Image-based rendering: A new interface between computer vision and computer graphics

W h a t is IBR? Image-based rendering (IBR) describes a set of techniques that al low three-dimensional graphical inter'action with objects and scenes whose original specification began as images or photographs. In an IBR pipeline, processing is applied to a set of input photographs creating an intermediate data structure. Later, this data structure is used to create new images of the scene or object. Computer vision provides tools to analyze Images and create models representing shape and surface properties. Computer graphics provides cools to take models and create images. This suggests that image-based rendering can be achieved by combining computer vision and computer graphics techniques. However, is combining the bestknown computer vision and graphics techniques the best we can do? Many computer vision algorithms are just not very robust, and high quality rendering can be compu=tional!y expensive, To better explore the connection between computer graphics and computer vision we ask the fol lowing question. How concise a model should one t ry to create from the image data using computer vision analysis? At one extreme, one could ask for a smooth surface representa t ion annotated w i th shading parameters.A less concise and unified representat ion is s imply a set of depth images, where a depth (z) value is assodar_=cl with each pixel of the original photographs. An even less concise representation associates with each image pixel a color value and a ~y direction. Concise representations are flexible. They permi t s t ra ight forward manipulat ions of models, including modifying shape and material properties. In addition, they al low for measurements and other forms of geometric reasoning (i.e. collision and proximity detect ion) . However, as we demand a more concise representation, the analysis becomes more difficult and the associated computer vision algorithms less reliable. Furthermore, more computation is required to synthesize images from these more concise representations. This leads us to the following questions. Is the transformation of images to the most concise representation possible a requirement for generating new images? Is a threedimensional model the best way of maintaining both the realism and integrity of the source images? Is a model merely a form of compression, for that matter, Iossy compression? Are we willing to tolerate such losses? Image-based rendering approaches threedimensional graphics problems by designing data st ructures that can be robust ly computed from images and can subsequently be used to create high quali ty images at minimal computat ional cost. Thus, imagebased rendering forces us to think about how to best use computer vision and computer graphics concepts and cools in conjunction with each other,