Building Stereoscopic Zoomer via Global and Local Warping Optimization

Magnifying left and right images by increasing focal length of optical zoom lenses is a direct way for stereoscopic zoom. However, without adjusting the baseline distance, the optical zoom may distort the three-dimensional (3D) perception. To tackle this problem, this article presents a stereoscopic zoomer that directly operates on image plane by synthesizing a virtual camera in depth direction to simulate the zoom process. Specifically, we propose a two-stage warping optimization framework for zooming, which is solved by imposing global and local energy constraints. Different from the existing 3D rendering methods, the important innovation of our method is to take content zoom, depth adaptation and user editability into account in mesh optimization. Experimental results demonstrate that our method can achieve promising zoom effect on daily person scenes and street views.

[1]  Peter H. N. de With,et al.  Free-viewpoint depth image based rendering , 2010, J. Vis. Commun. Image Represent..

[2]  Gangyi Jiang,et al.  StereoEditor: controllable stereoscopic display by content retargeting , 2017 .

[3]  Hongyang Chao,et al.  MeshStereo: A Global Stereo Model with Mesh Alignment Regularization for View Interpolation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[4]  David M. Hoffman,et al.  The zone of comfort: Predicting visual discomfort with stereo displays. , 2011, Journal of vision.

[5]  James D. K. Kim,et al.  Computational stereoscopic zoom , 2012 .

[6]  Vladlen Koltun,et al.  Zoom to Learn, Learn to Zoom , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Xi Wang,et al.  High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth , 2014, GCPR.

[8]  Xuming He,et al.  Geometry-Aware Deep Network for Single-Image Novel View Synthesis , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[9]  Yun Zhang,et al.  StereoPasting: Interactive Composition in Stereoscopic Images , 2013, IEEE Transactions on Visualization and Computer Graphics.

[10]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[11]  Feng Liu,et al.  3D Ken Burns effect from a single image , 2019, ACM Trans. Graph..

[12]  Kaiqi Huang,et al.  A2-RL: Aesthetics Aware Reinforcement Learning for Image Cropping , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[13]  Wu-chi Feng,et al.  Aesthetics-Based Stereoscopic Photo Cropping for Heterogeneous Displays , 2012, IEEE Transactions on Multimedia.

[14]  Gangyi Jiang,et al.  QoE-Guided Warping for Stereoscopic Image Retargeting , 2017, IEEE Transactions on Image Processing.

[15]  Ralph R. Martin,et al.  Manipulating Perspective in Stereoscopic Images , 2012 .

[16]  Graham R Jones,et al.  Controlling perceived depth in stereoscopic images , 2001, IS&T/SPIE Electronic Imaging.

[17]  Mei Yu,et al.  Three-dimensional visual comfort assessment via preference learning , 2015, J. Electronic Imaging.

[18]  Feng Shao,et al.  MSTGAR: Multioperator-Based Stereoscopic Thumbnail Generation With Arbitrary Resolution , 2020, IEEE Transactions on Multimedia.

[19]  Jian Shi,et al.  Image Retargeting Using Mesh Parametrization , 2009, IEEE Transactions on Multimedia.

[20]  Kwan-Liu Ma,et al.  Stereoscopic Thumbnail Creation via Efficient Stereo Saliency Detection , 2017, IEEE Transactions on Visualization and Computer Graphics.

[21]  Wei Zhang,et al.  ZoomNet: Part-Aware Adaptive Zooming Neural Network for 3D Object Detection , 2020, AAAI.

[22]  Michael F. Cohen,et al.  Panning and Zooming High-Resolution Panoramas in Virtual Reality Devices , 2017, UIST.

[23]  Xuelong Li,et al.  Shape-Preserving Object Depth Control for Stereoscopic Images , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[24]  Jianjun Lei,et al.  Projection-based disparity control for toed-in multiview images. , 2014, Optics express.

[25]  Pablo César,et al.  Efficient Depth-aware Image Deformation Adaptation for Curved Screen Displays , 2017, ACM Multimedia.

[26]  Rastislav Lukac,et al.  Digital camera zooming based on unified CFA image processing steps , 2004, IEEE Transactions on Consumer Electronics.

[27]  Noah Snavely,et al.  Single-View View Synthesis With Multiplane Images , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Sylvain Duchêne,et al.  Adapting stereoscopic movies to the viewing conditions using depth-preserving and artifact-free novel view synthesis , 2011, Electronic Imaging.

[29]  Paul S. Heckbert,et al.  Fundamentals of Texture Mapping and Image Warping , 1989 .

[30]  Yusheng Ji,et al.  On Constructing $z$-Dimensional DIBR-Synthesized Images , 2016, IEEE Transactions on Multimedia.

[31]  Dong Hyun Kim,et al.  Convergence Point Adjustment Improving Visual Discomfort for a Zoom on a Stereoscopic Camera , 2016 .

[32]  Jérôme Gensel,et al.  Content zooming and exploration for mobile maps , 2012, GIScience 2012.

[33]  Yo-Sung Ho,et al.  Roundness-Preserving Warping for Aesthetic Enhancement-based Stereoscopic Image Editing , 2020 .

[34]  Yung-Yu Chuang,et al.  Scene warping: Layer-based stereoscopic image resizing , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[35]  Michael J. Black,et al.  Secrets of optical flow estimation and their principles , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[36]  Wen-Huang Cheng,et al.  Perspective-aware warping for seamless stereoscopic image cloning , 2012, ACM Trans. Graph..

[37]  Graham Fyffe,et al.  Stereo Magnification: Learning View Synthesis using Multiplane Images , 2018, ArXiv.

[38]  Kok-Lim Low,et al.  Aesthetics-Driven Stereoscopic 3-D Image Recomposition With Depth Adaptation , 2018, IEEE Transactions on Multimedia.

[39]  Kok-Lim Low,et al.  Stereoscopic image warping for enhancing composition aesthetics , 2015, 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR).

[40]  Yung-Yu Chuang,et al.  Content-Aware Display Adaptation and Interactive Editing for Stereoscopic Images , 2011, IEEE Transactions on Multimedia.

[41]  Christoph Fehn,et al.  Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV , 2004, IS&T/SPIE Electronic Imaging.

[42]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Peter Eisert,et al.  The Stereoscopic Analyzer — An image-based assistance tool for stereo shooting and 3D production , 2010, 2010 IEEE International Conference on Image Processing.

[44]  Makoto Okui,et al.  Geometrical analysis of puppet-theater and cardboard effects in stereoscopic HDTV images , 2006, IEEE Transactions on Circuits and Systems for Video Technology.

[45]  Pascal Frossard,et al.  Graph-based interpolation for zooming in 3D scenes , 2017, 2017 25th European Signal Processing Conference (EUSIPCO).

[46]  Shi-Min Hu,et al.  Aesthetic Image Enhancement by Dependence-Aware Object Recomposition , 2013, IEEE Transactions on Multimedia.

[47]  Rynson W. H. Lau,et al.  Depth Mapping for Stereoscopic Videos , 2013, International Journal of Computer Vision.

[48]  Juan Luis Gonzalez Bello,et al.  Deep 3D-Zoom Net: Unsupervised Learning of Photo-Realistic 3D-Zoom , 2019, ArXiv.

[49]  Qionghai Dai,et al.  Toward Simultaneous Visual Comfort and Depth Sensation Optimization for Stereoscopic 3-D Experience , 2017, IEEE Transactions on Cybernetics.

[50]  M. Gross,et al.  Nonlinear disparity mapping for stereoscopic 3D , 2010, ACM Trans. Graph..

[51]  Shi-Min Hu,et al.  PhotoRecomposer: Interactive Photo Recomposition by Cropping , 2018, IEEE Transactions on Visualization and Computer Graphics.

[52]  Wei Song,et al.  Impact of zooming and enhancing region of interests for optimizing user experience on mobile sports video , 2010, ACM Multimedia.

[53]  Sanghoon Sull,et al.  Efficient Viewer-Centric Depth Adjustment Based on Virtual Fronto-Parallel Planar Projection in Stereo 3D Images , 2014, IEEE Transactions on Multimedia.

[54]  Rémi Ronfard,et al.  The Stereoscopic Zoom , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).