Non-photorealistic camera: depth edge detection and stylized rendering using multi-flash imaging

We present a non-photorealistic rendering approach to capture and convey shape features of real-world scenes. We use a camera with multiple flashes that are strategically positioned to cast shadows along depth discontinuities in the scene. The projective-geometric relationship of the camera-flash setup is then exploited to detect depth discontinuities and distinguish them from intensity edges due to material discontinuities.We introduce depiction methods that utilize the detected edge features to generate stylized static and animated images. We can highlight the detected features, suppress unnecessary details or combine features from multiple images. The resulting images more clearly convey the 3D structure of the imaged scenes.We take a very different approach to capturing geometric features of a scene than traditional approaches that require reconstructing a 3D model. This results in a method that is both surprisingly simple and computationally efficient. The entire hardware/software setup can conceivably be packaged into a self-contained device no larger than existing digital cameras.

[1]  Ramesh Raskar,et al.  Exploiting Depth Discontinuities for Vision-Based Fingerspelling Recognition , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[2]  Thomas S. Huang,et al.  Image processing , 1971 .

[3]  Pat Hanrahan,et al.  Conveying shape and features with image-based relighting , 2003, IEEE Visualization, 2003. VIS 2003..

[4]  Yoshiaki Shirai,et al.  Extraction of the Line Drawings of 3-Dimensional Objects by Sequential Illumination From Several Directions , 1971, International Joint Conference on Artificial Intelligence.

[5]  Pietro Perona,et al.  Shadow Carving , 2001, ICCV.

[6]  Frédo Durand,et al.  An invitation to discuss computer depiction , 2002, NPAR '02.

[7]  Stan Birchfield,et al.  Depth and motion discontinuities , 1999 .

[8]  Dani Lischinski,et al.  Gradient Domain High Dynamic Range Compression , 2023 .

[9]  William H. Press,et al.  Numerical recipes in C. The art of scientific computing , 1987 .

[10]  Stefan Schlechtweg,et al.  Non-photorealistic computer graphics: modeling, rendering, and animation , 2002 .

[11]  Scott F. Johnston.  Lumo: illumination for cel animation , 2002, NPAR '02.

[12]  Bruce Gooch,et al.  Non-photorealistic rendering , 2001 .

[13]  Yair Weiss,et al.  Deriving intrinsic images from image sequences , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[14]  Ramesh Raskar,et al.  Shape-Enhanced Surgical Visualizations and Medical Illustrations with Multi-flash Imaging , 2004, MICCAI.

[15]  Steven W. Zucker,et al.  Finding folds: on the appearance and identification of occlusion , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[16]  Gregory Dudek,et al.  Space occupancy using multiple shadowimages , 1995, Proceedings 1995 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human Robot Interaction and Cooperative Robots.

[17]  Adam Finkelstein,et al.  Suggestive contours for conveying shape , 2003, ACM Trans. Graph..

[18]  Ramesh Raskar,et al.  Image fusion for context enhancement and video surrealism , 2004, NPAR '04.

[19]  Alan L. Yuille,et al.  Occlusions and Binocular Stereo , 1992, ECCV.

[20]  Takafumi Saito,et al.  Comprehensible rendering of 3-D shapes , 1990, SIGGRAPH.

[21]  Stephen Lin,et al.  Highlight removal by illumination-constrained inpainting , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[22]  Aaron Hertzmann,et al.  Painterly rendering with curved brush strokes of multiple sizes , 1998, SIGGRAPH.

[23]  R. Bruce Irvin,et al.  Methods For Exploiting The Relationship Between Buildings And Their Shadows In Aerial Imagery , 1989, Photonics West - Lasers and Applications in Science and Engineering.

[24]  David Salesin,et al.  Shadow matting and compositing , 2003, ACM Trans. Graph..

[25]  J. Kender,et al.  Shape from darkness under error , 1996 .

[26]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[27]  Kenneth A. Loparo,et al.  Reconstruction of three-dimensional surfaces from two-dimensional binary images , 1989, IEEE Trans. Robotics Autom..

[28]  Ramakant Nevatia,et al.  Building Detection and Description from a Single Intensity Image , 1998, Comput. Vis. Image Underst..

[29]  Gregory Dudek,et al.  On 3-D surface reconstruction using shape from shadows , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[30]  Ramakant Nevatia,et al.  Detecting buildings in aerial images , 1988, Comput. Vis. Graph. Image Process..

[31]  Douglas DeCarlo,et al.  Stylization and abstraction of photographs , 2002, ACM Trans. Graph..

[32]  Xenophon Papademetris,et al.  Estimation of motion boundary location and optical flow using dynamic programming , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[33]  Jean Ponce,et al.  Computer Vision: A Modern Approach , 2002 .

[34]  Kentaro Toyama,et al.  Wallflower: principles and practice of background maintenance , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[35]  James H. Elder,et al.  Are Edges Incomplete? , 1999, International Journal of Computer Vision.

[36]  David J. Kriegman,et al.  What shadows reveal about object structure. , 2001 .

[37]  Katsushi Ikeuchi,et al.  Stability issues in recovering illumination distribution from brightness in shadows , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[38]  William H. Press,et al.  The Art of Scientific Computing Second Edition , 1998 .

[39]  Richard Szeliski,et al.  Handling occlusions in dense multi-view stereo , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[40]  R. Bruce Irvin,et al.  Methods for exploiting the relationship between buildings and their shadows in aerial imagery , 1989, IEEE Trans. Syst. Man Cybern..