Structure Models for Image-Assisted Geometry Measurement in Plenoptic Sampling

We present a signal-processing framework for image-assisted geometry measurement in the image-based rendering (IBR). We study the utilized geometry information and estimating minimum sampling rate of the IBR. Our method combines decomposing a complex scene geometry into a collection of simpler structures on a block-by-block basis. The automatic simpler structure selection can be interactively refined by detected single salient points. In this manner, we reduce the spectral analysis problem of an irregular object to that of a simpler structure. Predictions on the frequency content can then be used to control the sampling rate. This extends previous work in which the IBR sampling is analyzed and estimated for nonuniform sampling. Extensive experimental evaluation demonstrates that our geometry simplification method significantly outperforms competing algorithms. Additionally, the minimum sampling rate of the IBR necessary for alias-free rendering will be reduced as the number of simpler structures increases.

[1]  Li Yu,et al.  Image-assisted geometry simplification for the plenoptic sampling , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[3]  Robert Bregovic,et al.  Light Field Reconstruction Using Shearlet Transform , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Ralph R. Martin,et al.  A Shape‐Preserving Approach to Image Resizing , 2009, Comput. Graph. Forum.

[5]  Heiko Schwarz,et al.  Partial Depth Image Based Re-Rendering for Synthesized View Distortion Computation , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[6]  Yu-Wing Tai,et al.  Automatic Trimap Generation and Consistent Matting for Light-Field Images. , 2017, IEEE transactions on pattern analysis and machine intelligence.

[7]  Homer H. Chen,et al.  Light Field Analysis for Modeling Image Formation , 2011, IEEE Transactions on Image Processing.

[8]  Mike Brookes,et al.  Plenoptic Layer-Based Modeling for Image Based Rendering , 2013, IEEE Transactions on Image Processing.

[9]  Harry Shum,et al.  On the number of samples needed in light field rendering with constant-depth assumption , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[10]  Ichiro Yuyama,et al.  A study on resolution and aliasing for multi-viewpoint image acquisition , 2000, IEEE Trans. Circuits Syst. Video Technol..

[11]  Wanqing Li,et al.  Estimation of Signal Distortion Using Effective Sampling Density for Light Field-Based Free Viewpoint Video , 2015, IEEE Transactions on Multimedia.

[12]  George Drettakis,et al.  Silhouette‐Aware Warping for Image‐Based Rendering , 2011, Comput. Graph. Forum.

[13]  Julia Navarro,et al.  Robust and Dense Depth Estimation for Light Field Images , 2017, IEEE Transactions on Image Processing.

[14]  P. Belhumeur,et al.  Moving gradients: a path-based method for plausible image interpolation , 2009, SIGGRAPH 2009.

[15]  Donald S. Fussell,et al.  Uniformly Sampled Light Fields , 1998, Rendering Techniques.

[16]  Minh N. Do,et al.  Ieee Transactions on Image Processing on the Bandwidth of the Plenoptic Function , 2022 .

[17]  J. Berent,et al.  Plenoptic Manifolds , 2007, IEEE Signal Processing Magazine.

[18]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[19]  Frédo Durand,et al.  Light Field Reconstruction Using Sparsity in the Continuous Fourier Domain , 2014, ACM Trans. Graph..

[20]  Janusz Grzyb,et al.  Terahertz Light-Field Imaging , 2016, IEEE Transactions on Terahertz Science and Technology.

[21]  Ana-Maria Cretu,et al.  Static and Dynamic Hand Gesture Recognition in Depth Data Using Dynamic Time Warping , 2016, IEEE Transactions on Instrumentation and Measurement.

[22]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Penglang Shui,et al.  Corner Detection and Classification Using Anisotropic Directional Derivative Representations , 2013, IEEE Transactions on Image Processing.

[24]  Robert C. Bolles,et al.  Epipolar-plane image analysis: An approach to determining structure from motion , 1987, International Journal of Computer Vision.

[25]  Leo Van Biesen,et al.  OPTIMAL MODEL FOR THE DIFFRACTION EFFECT IN THE ULTRASONIC FIELD OF PISTON TRANSDUCERS , 2001 .

[26]  F. Durand A Frequency Analysis of Light Transport , 2011 .

[27]  Jed Lengyel,et al.  The Convergence of Graphics and Vision , 1998, Computer.

[28]  George Drettakis,et al.  Scalable inside-out image-based rendering , 2016, ACM Trans. Graph..

[29]  Harry Shum,et al.  Plenoptic sampling , 2000, SIGGRAPH.

[30]  Otmar Loffeld,et al.  Adaptive High Dynamic Range for Time-of-Flight Cameras , 2015, IEEE Transactions on Instrumentation and Measurement.

[31]  In-So Kweon,et al.  Geometric Calibration of Micro-Lens-Based Light Field Cameras Using Line Features , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Ravi Ramamoorthi,et al.  A Fourier Theory for Cast Shadows , 2004, ECCV.

[33]  Adrian G. Bors,et al.  Correcting 3D scenes estimated from sets of multi-view images using shape-from-contours , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[34]  Boris Beizer,et al.  Black Box Testing: Techniques for Functional Testing of Software and Systems , 1996, IEEE Software.

[35]  Mark Hedley,et al.  Fast corner detection , 1998, Image Vis. Comput..

[36]  Leif Kobbelt,et al.  Interactive Pixel‐Accurate Free Viewpoint Rendering from Images with Silhouette Aware Sampling , 2009, Comput. Graph. Forum.

[37]  Leonard McMillan,et al.  A new reconstruction filter for undersampled light fields , 2003 .

[38]  Harry Shum,et al.  Rendering with concentric mosaics , 1999, SIGGRAPH.

[39]  Takeshi Naemura,et al.  Layered light-field rendering with focus measurement , 2006, Signal Process. Image Commun..

[40]  S. B. Kang,et al.  Survey of image-based representations and compression techniques , 2003, IEEE Trans. Circuits Syst. Video Technol..

[41]  Robert Bregovic,et al.  Fixed-Pattern Noise Modeling and Removal in Time-of-Flight Sensing , 2016, IEEE Transactions on Instrumentation and Measurement.

[42]  Mike Brookes,et al.  On the Spectrum of the Plenoptic Function , 2014, IEEE Transactions on Image Processing.

[43]  Glenn Healey,et al.  Analyzing the bidirectional texture function , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[44]  Tsuhan Chen,et al.  Spectral analysis for sampling image-based rendering data , 2003, IEEE Trans. Circuits Syst. Video Technol..

[45]  Karthikeyan Vaidyanathan,et al.  Layered Light Field Reconstruction for Defocus Blur , 2015, TOGS.

[46]  Jitendra Malik,et al.  Shape Estimation from Shading, Defocus, and Correspondence Using Light-Field Angular Coherence , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[48]  Harry Shum,et al.  Pop-up light field: An interactive image-based modeling and rendering system , 2004, TOGS.

[49]  Wanqing Li,et al.  Objective evaluation of light field rendering methods using effective sampling density , 2011, 2011 IEEE 13th International Workshop on Multimedia Signal Processing.

[50]  Marc Levoy,et al.  Light Fields and Computational Imaging , 2006, Computer.

[51]  Harry Shum,et al.  Am object-based approach to plenoptic videos , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[52]  M.N. Do,et al.  Error Analysis for Image-Based Rendering With Depth Information , 2006, IEEE Transactions on Image Processing.

[53]  M. Landy,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[54]  Michael P. Fitz,et al.  Further results in the fast estimation of a single frequency , 1994, IEEE Trans. Commun..