Reading between the lines—a method for extracting dynamic 3D with texture

A method is presented that extracts the 3D shape of objects, together with their surface texture. Both shape and texture are obtained from a single image. The paper sketches the complete system but focuses on the problem of texture extraction. The underlying principle is based on an active technique. A high resolution pattern is projected onto the object and the deformations as observed by a single camera yield the 3rd dimension. The surface texture is extracted from the same image by literally reading between the lines that are used for the shape extraction. This is done using a combination of interpolation and non-linear diffusion techniques. Because the whole procedure is based on a single image, a frame-byframe reconstruction of a video taken with the pattern projected throughout, yields 3D shape dynamics. 1 A paradigm shift in 3D Most methods for three-dimensional shape extraction go via the explicit calculation of the distance between the sensor and the object. The 3D shape of the surface then follows from the variation of this distance. This is what 3D acquisition devices for reverse engineering, shape inspection, and mobile robotics wotild typically do. More recent applications, however, focus on visualization, such as virtual and augmented reality, 3D on the Internet, special effects in movies, etc. These kind of applications come with different priorities: l Extracting the absolute scale of objects usually is not crucial. Objects. will be shown at completely different scales anyway. Permission to make digitalhud copies ofall or pat ofthis material for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage, the copyright notice, the title of the publication and its date appear, and notice is given that copyright is by pemhsion ofthe ACM, Inc. To copy oh&se, to republish, to post on servers or to redistribute to lists, requires specific permission and/or fee / ACM KWT ‘97 Lausanne Switzerland Copyright 1997 ACM 0-89791-953497/9..$3.50 l The extraction of surface texture becomes a prerequisite. Traditional 3D acquisition systems often lack this ability to align the shape and texture data. l In the top range of visualisation applications such as the movies and virtual worlds 3D dynamics becomes more important. Rather than building a still model and animating it via off-line motion tracking, direct extraction of 3D motion would strongly alleviate such tasks. l Also, with Internet at the fingertips of users in small companies and amateurs at home, 3D acquisition devices should become much cheaper, less bulky and easier to use in order for 3D models to fully pen& trate the Net. The paper proposes a novel, 3D acquisition system that is geared towards these new requirements, i.e. the extraction of 3D models for visualisation. The hardware needed is a simple slide projector, a normal camera, and a computer. Setting up tlie system is easy and requires no exotic calibration objects. It uses special illumination to obtain good geometric precision. From a single image, both 3D shape and surface texture are extracted. This turns the system into a say 4D scanner as it becomes possible to extract dynamic 3D by frame by frame reconstruction of video data. A complete description of the system is outside the scope of this short paper. This paper will mainly focus on the aspect of surface texture extraction (section 3). Nevertheless, section 2 will sketch how the 3D shape is extracted. Results of 3D shapes together with the texture and 3D motions are shown in section 4. That section also shows a preliminary example of how such data can be used for special effects or animation. Section 5 concludes the paper. 2 One-shot 3D acquisition 2.1 Comparison with other methods As passive 3D acquisition systems often lack precision, certainly in untextured areas, we opted for an active ap preach, where a simple square pattern is projected on the