The visual computing of projector-camera systems

This article focuses on real-time image correction techniques that enable projector-camera systems to display images onto screens that are not optimized for projections, such as geometrically complex, colored and textured surfaces. It reviews hardware accelerated methods like pixel-precise geometric warping, radiometric compensation, multi-focal projection, and the correction of general light modulation effects. Online and offline calibration as well as invisible coding methods are explained. Novel attempts in super-resolution, high dynamic range and high-speed projection are discussed. These techniques open a variety of new applications for projection displays. Some of them will also be presented in this report.

[1]  Shree K. Nayar,et al.  Programmable imaging using a digital micromirror array , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[2]  Oliver Bimber,et al.  Embedded entertainment with smart projectors , 2005, Computer.

[3]  W. Brent Seales,et al.  Dynamic shadow removal from front projection displays , 2001, Proceedings Visualization, 2001. VIS '01..

[4]  O. Bimber,et al.  Superimposing pictorial artwork with projected imagery , 2004, SIGGRAPH Courses.

[5]  Greg Welch,et al.  The office of the future: a unified approach to image-based modeling and spatially immersive displays , 1998, SIGGRAPH.

[6]  Robert Ulichney,et al.  47.4: Invited Paper: Wobulation: Doubling the Addressed Resolution of Projection Displays , 2005 .

[7]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, ACM Trans. Graph..

[8]  Shree K. Nayar,et al.  A projector-camera system with real-time photometric adaptation for dynamic environments , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[9]  Markus H. Gross,et al.  Embedding imperceptible patterns into projected images for simultaneous acquisition and display , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[10]  P. Belhumeur,et al.  Making one object look like another: controlling appearance using a projector-camera system , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[11]  Oliver Bimber,et al.  Compensating Indirect Scattering for Immersive and Semi-Immersive Projection Displays , 2006, IEEE Virtual Reality Conference (VR 2006).

[12]  Jong-Il Park,et al.  Specularity-Free Projection on Nonplanar Surface , 2005, PCM.

[13]  P. Debevec,et al.  Rendering for an interactive 360° light field display , 2007, ACM Trans. Graph..

[14]  Oliver Bimber,et al.  Real-Time Adaptive Radiometric Compensation , 2008, IEEE Trans. Vis. Comput. Graph..

[15]  Christopher Jaynes,et al.  Super-Resolution Composition in Multi-Projector Displays , 2003 .

[16]  Ruigang Yang,et al.  Automatic and Continuous Projector Display Surface Estimation Using Everyday Imagery , 2001, WSCG.

[17]  Roland Hoefling,et al.  ALP: universal DMD controller for metrology and testing , 2004, IS&T/SPIE Electronic Imaging.

[18]  Markus H. Gross,et al.  Scalable 3D video of dynamic scenes , 2005, The Visual Computer.

[19]  Andriy Pavlovych,et al.  A high-dynamic range projection system , 2005, Photonics North.

[20]  Hideshi Yamada,et al.  Rendering for an interactive 360° light field display , 2007, ACM Trans. Graph..

[21]  Darren Leigh,et al.  RFIG lamps: interacting with a self-describing world via photosensing wireless tags and projectors , 2004, SIGGRAPH 2004.

[22]  Shree K. Nayar,et al.  Projection defocus analysis for scene capture and image display , 2006, SIGGRAPH 2006.

[23]  Aditi Majumder,et al.  Geometric Modeling and Calibration of Planar Multi-Projector Displays Using Rational Bezier Patches , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Ruigang Yang,et al.  Camera-based calibration techniques for seamless multiprojector displays , 2005, IEEE Transactions on Visualization and Computer Graphics.

[25]  Narendra Ahuja,et al.  Split Aperture Imaging for High Dynamic Range , 2004, International Journal of Computer Vision.

[26]  Walter M. Duncan,et al.  Emerging digital micromirror device (DMD) applications , 2003, SPIE MOEMS-MEMS.

[27]  Joaquim Salvi,et al.  Pattern codification strategies in structured light systems , 2004, Pattern Recognit..

[28]  Shree K. Nayar,et al.  A Projector-Camera System with Real-Time Photometric Adaptation for Dynamic Environments , 2005, CVPR.

[29]  Hideo Saito,et al.  Focal Pre-Correction of Projected Image for Deblurring Screen Image , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Niranjan Damera-Venkata,et al.  Realizing Super-Resolution with Superimposed Projection , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Jong-Il Park,et al.  Contrast Enhancement in Direct-Projected Augmented Reality , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[32]  Scott S. Fisher,et al.  Snared illumination , 2004, SIGGRAPH '04.

[33]  T. Yoshida,et al.  A Virtual Color Reconstruction System for Real Heritage with Light Projection , 2003 .

[34]  James M. Rehg,et al.  Projector-guided painting , 2006, UIST.

[35]  Erik Reinhard,et al.  High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting (The Morgan Kaufmann Series in Computer Graphics) , 2005 .

[36]  Oliver Bimber,et al.  Real-Time Adaptive Radiometric Compensation , 2006, IEEE Transactions on Visualization and Computer Graphics.

[37]  Donald P. Greenberg,et al.  A perceptually based physical error metric for realistic image synthesis , 1999, SIGGRAPH.

[38]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[39]  Ramesh Raskar,et al.  Fast separation of direct and global components of a scene using high frequency illumination , 2006, SIGGRAPH 2006.

[40]  Aditi Majumder,et al.  Photometric Self-Calibration of a Projector-Camera System , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[41]  Joaquim Salvi,et al.  Recent progress in coded structured light as a technique to solve the correspondence problem: a survey , 1998, Pattern Recognit..

[42]  Oliver Bimber,et al.  Passive-Active Geometric Calibration for View-Dependent Projections onto Arbitrary Surfaces , 2007, J. Virtual Real. Broadcast..

[43]  Moon Gi Kang,et al.  Super-resolution image reconstruction: a technical overview , 2003, IEEE Signal Process. Mag..

[44]  Aditi Majumder,et al.  Registration Techniques for Using Imperfect and Par tially Calibrated Devices in Planar Multi-Projector Displays , 2007, IEEE Transactions on Visualization and Computer Graphics.

[45]  Mark Ashdown,et al.  Robust Content-Dependent Photometric Projector Compensation , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[46]  Takeshi Naemura,et al.  A Study on Position-dependent Visible Light Communication using DMD for ProCam , 2006 .

[47]  Gita Reese Sukthankar,et al.  Dynamic shadow elimination for multi-projector displays , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[48]  Joseph Shamir,et al.  Range Imaging With Adaptive Color Structured Light , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[49]  Shree K. Nayar,et al.  A Projection System with Radiometric Compensation for Screen Imperfections , 2003 .

[50]  Gordon Wetzstein,et al.  Enabling view-dependent stereoscopic projection in real environments , 2005, SIGGRAPH '05.

[51]  Erik Reinhard,et al.  High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting , 2010 .

[52]  Jong-Il Park,et al.  Radiometrically-Compensated Projection onto Non-Lambertian Surface Using Multiple Overlapping Projectors , 2006, PSIVT.

[53]  Oliver Bimber,et al.  Dynamic Adaptation of Projected Imperceptible Codes , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[54]  Shree K. Nayar,et al.  Adaptive dynamic range imaging: optical control of pixel exposures over space and time , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[55]  Wolfgang Heidrich,et al.  High dynamic range display systems , 2004, ACM Trans. Graph..

[56]  Markus H. Gross,et al.  Adaptive Instant Displays: Continuously Calibrated Projections Using Per‐Pixel Light Control , 2005, Comput. Graph. Forum.

[57]  Oliver Bimber,et al.  Imperceptible Calibration for Radiometric Compensation , 2007, Eurographics.

[58]  Marc Levoy,et al.  High-speed videography using a dense camera array , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[59]  Wolfgang Heidrich,et al.  3.2: High Dynamic Range Projection Systems , 2007 .

[60]  Claudio S. Pinhanez.  Using a steerable projector and a camera to transform surfaces into interactive displays , 2001, CHI Extended Abstracts.

[61]  Ramesh Raskar,et al.  Quadric Transfer for Immersive Curved Screen Displays , 2004, Comput. Graph. Forum.

[62]  Takahiro Okabe,et al.  Radiometric Compensation in a Projector-Camera System Based Properties of Human Vision System , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[63]  Kiriakos N. Kutulakos,et al.  A theory of inverse light transport , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[64]  Yuichi Ohta,et al.  Analytical compensation of inter-reflection for pattern projection , 2006, VRST '06.

[65]  Ramesh Raskar,et al.  Oblique projector rendering on planar surfaces for a tracked user , 1999, SIGGRAPH '99.

[66]  Henry Fuchs,et al.  Real-Time Projector Tracking on Complex Geometry Using Ordinary Imagery , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[67]  Oliver Bimber,et al.  Embedded Entertainment with Smart Projectors , 2005, Computer.

[68]  Oliver Bimber,et al.  Multifocal projection: a multiprojector technique for increasing focal depth , 2006, IEEE Transactions on Visualization and Computer Graphics.

[69]  Wolfgang Heidrich,et al.  High dynamic range display systems , 2004, SIGGRAPH 2004.

[70]  S. Nayar,et al.  Projection defocus analysis for scene capture and image display , 2006, ACM Trans. Graph..

[71]  Gordon Wetzstein,et al.  Radiometric Compensation through Inverse Light Transport , 2007, 15th Pacific Conference on Computer Graphics and Applications (PG'07).

[72]  Ramesh Raskar,et al.  Prakash: lighting aware motion capture using photosensing markers and multiplexed illuminators , 2007, SIGGRAPH 2007.

[73]  W. Brent Seales,et al.  Multi-projector displays using camera-based registration , 1999, Proceedings Visualization '99 (Cat. No.99CB37067).

[74]  Oliver Bimber,et al.  Projector-Based Augmentation , 2007 .

[75]  Gordon Wetzstein,et al.  Coded aperture projection , 2008, SIGGRAPH '08.

[76]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, SIGGRAPH 2005.

[77]  Jong-Il Park,et al.  Content Adaptive Embedding of Complementary Patterns for Nonintrusive Direct-Projected Augmented Reality , 2007, HCI.

[78]  Andrew Gardner,et al.  Simulating spatially varying lighting on a live performance , 2006 .

[79]  Michitaka Hirose,et al.  Projected augmentation - augmented reality using rotatable video projectors , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[80]  Masahiko Inami,et al.  Smart Light-Ultra High Speed Projector for Spatial Multiplexing Optical Transmission , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[81]  Paul A. Beardsley,et al.  RFIG lamps: interacting with a self-describing world via photosensing wireless tags and projectors , 2004, ACM Trans. Graph..

[82]  Koichi Hashimoto,et al.  1P1-M02 Development of a 3,000-fps 3D Imaging System Using a High-Speed Projector , 2007 .

[83]  Steve Marschner,et al.  Dual photography , 2005, ACM Trans. Graph..

[84]  Hitoshi Habe,et al.  Inter-Reflection Compensation for Immersive Projection Display , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[85]  Christopher O. Jaynes,et al.  Camera-based detection and removal of shadows from interactive multiprojector displays , 2004, IEEE Transactions on Visualization and Computer Graphics.

[86]  Ramesh Raskar,et al.  Prakash: lighting aware motion capture using photosensing markers and multiplexed illuminators , 2007, ACM Trans. Graph..

[87]  Takahiro Okabe,et al.  Perceptual Photometric Compensation for Projected Images , 2006 .

[88]  Michael S. Brown,et al.  Image Pre-Conditioning for Out-of-Focus Projector Blur , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[89]  Oliver Bimber,et al.  Superimposing pictorial artwork with projected imagery , 2005, IEEE MultiMedia.

[90]  Luiz Velho,et al.  A Camera-Projector System for Real-Time 3D Video , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[91]  Greg Welch,et al.  COMPUTER GRAPHICS OPTIQUE Optical Superposition of Projected Computer Graphics , 2001, EGVE/IPT.

[92]  Koichi Hashimoto,et al.  3,000-fps 3-D shape measurement using a high-speed camera-projector system , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.