From stereo image sequences to smooth and robust surface models using temporal information and Bilateral postprocessing

Reconstruction of surface models from camera images has many applications in robotics such as surface registration or object recognition. In this paper, we describe a workflow in which we extract depth information from stereo image sequences to generate a surface model. We present our solutions to correspondence analysis, disparity correction and refinement, as well as 3D reconstruction, point cloud smoothing and meshing. One important feature of the correspondence analysis that we evaluate in detail is the use of temporal information. Another emphasis is on correcting and smoothing the disparity images as well as the reconstructed point cloud without losing too much detail. We, hence, introduce our application of the Bilateral filter on disparity images and our usage of least squares smoothing. The components of the workflow were evaluated using three image sources: Endoscopic images from the daVinci® telemanipulator; images from a stereo camera integrated in the ARMAR III humanoid robot; synthetic data. Depending on the image resolution and the application, the workflow reconstructs surface models in real-time. We show that by using temporal information we obtain more accurate and robust correspondences. Additionally, the Bilateral filter was especially useful in refining the correspondences extracted from endoscopic images as well as the synthetic data sets, whereas the least squares method showed good results in smoothing the point cloud of ARMAR III images. Overall, the presented approach achieves good results for different camera settings and image types, especially with respect to the real-time requirement.