CompenNet++: End-to-End Full Projector Compensation

Full projector compensation aims to modify a projector input image such that it can compensate for both geometric and photometric disturbance of the projection surface. Traditional methods usually solve the two parts separately, although they are known to correlate with each other. In this paper, we propose the first end-to-end solution, named CompenNet++, to solve the two problems jointly. Our work non-trivially extends CompenNet, which was recently proposed for photometric compensation with promising performance. First, we propose a novel geometric correction subnet, which is designed with a cascaded coarse-to-fine structure to learn the sampling grid directly from photometric sampling images. Second, by concatenating the geometric correction subset with CompenNet, CompenNet++ accomplishes full projector compensation and is end-to-end trainable. Third, after training, we significantly simplify both geometric and photometric compensation parts, and hence largely improves the running time efficiency. Moreover, we construct the first setup-independent full compensation benchmark to facilitate the study on this topic. In our thorough experiments, our method shows clear advantages over previous arts with promising compensation quality and meanwhile being practically convenient.

[1]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  Shree K. Nayar,et al.  A projector-camera system with real-time photometric adaptation for dynamic environments , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[3]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[4]  Michael Harville,et al.  Practical Methods for Geometric and Photometric Correction of Tiled Projector , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[5]  Oliver Bimber,et al.  Embedded entertainment with smart projectors , 2005, Computer.

[6]  Gabriel Taubin,et al.  Simple, Accurate, and Robust Projector-Camera Calibration , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[7]  T. Yoshida,et al.  A Virtual Color Reconstruction System for Real Heritage with Light Projection , 2003 .

[8]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Josef Sivic,et al.  Convolutional Neural Network Architecture for Geometric Matching , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Greg Welch,et al.  Shader Lamps: Animating Real Objects With Image-Based Illumination , 2001, Rendering Techniques.

[11]  Anselm Grundhöfer,et al.  Robust, Error-Tolerant Photometric Projector Compensation , 2015, IEEE Transactions on Image Processing.

[12]  Shree K. Nayar,et al.  A Projection System with Radiometric Compensation for Screen Imperfections , 2003 .

[13]  Andrew L. Maas.  Rectifier Nonlinearities Improve Neural Network Acoustic Models , 2013 .

[14]  Serge J. Belongie,et al.  Approximate Thin Plate Spline Mappings , 2002, ECCV.

[15]  Aditi Majumder,et al.  ADICT: Accurate Direct and Inverse Color Transformation , 2010, ECCV.

[16]  Bingyao Huang,et al.  End-To-End Projector Photometric Compensation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Masatoshi Ishikawa,et al.  Dynamic Projection Mapping onto Deforming Non-Rigid Surface Using Deformable Dot Cluster Marker , 2017, IEEE Transactions on Visualization and Computer Graphics.

[18]  N. Otsu.  A threshold selection method from gray level histograms , 1979 .

[19]  Christian Siegl,et al.  Adaptive stray-light compensation in dynamic multi-projection mapping , 2017, Computational Visual Media.

[20]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[21]  Jan Kautz,et al.  Loss Functions for Image Restoration With Neural Networks , 2017, IEEE Transactions on Computational Imaging.

[22]  Bingyao Huang,et al.  A Single-Shot-Per-Pose Camera-Projector Calibration System for Imperfect Planar Targets , 2018, 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct).

[23]  Kosuke Sato,et al.  Fabricating Diminishable Visual Markers for Geometric Registration in Projection Mapping , 2018, IEEE Transactions on Visualization and Computer Graphics.

[24]  Paul A. Beardsley,et al.  A self-correcting projector , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[25]  Anselm Grundhöfer,et al.  Recent Advances in Projection Mapping Algorithms, Hardware and Applications , 2018, Comput. Graph. Forum.

[26]  Paul A. Beardsley,et al.  Natural video matting using camera arrays , 2006, ACM Trans. Graph..

[27]  David A. Clausi,et al.  Saliency-guided projection geometric correction using a projector-camera system , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[28]  Justus Thies,et al.  Real-time pixel luminance optimization for dynamic multi-projection mapping , 2015, ACM Trans. Graph..

[29]  Daniel G. Aliaga,et al.  Fast high-resolution appearance editing using superimposed projections , 2012, TOGS.

[30]  Sabine Süsstrunk,et al.  Simultaneous Geometric and Radiometric Calibration of a Projector-Camera Pair , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Kosuke Sato,et al.  Inter-reflection Compensation of Immersive Projection Display by Spatio-Temporal Screen Reflectance Modulation , 2016, IEEE Transactions on Visualization and Computer Graphics.

[32]  Sébastien Roy,et al.  Multi-projectors for arbitrary surfaces without explicit calibration nor reconstruction , 2003, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings..

[33]  Mark Ashdown,et al.  Robust Content-Dependent Photometric Projector Compensation , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[34]  Jason Geng,et al.  Structured-light 3D surface imaging: a tutorial , 2011 .

[35]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[36]  Chong Wang,et al.  Practical Radiometric Compensation for Projection Display on Textured Surfaces using a Multidimensional Model , 2018, Comput. Graph. Forum.

[37]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[38]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[39]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).