A Shape-Adaptive Low-Complexity Technique for 3D Free-Viewpoint Visual Applications

View synthesis (VS) and depth map generation (DMG) are two of the most crucial operations for facilitating the development of free-viewpoint 3D video systems. This demand for high computational complexity has motivated the development of the proposed design. Conventional designs have directly adopted dedicated hardware to increase the speed of these operations independently, which has resulted in algorithmic-level redundancies. Thus, the conventional per-pixel computation results in high redundancy in both VS and DMG operations. Accordingly, this paper proposes a shape-adaptive low-complexity (SALC) technique for removing such algorithmic-level redundancies for both VS and DMG operations. Moreover, adopting shape-adaptive searching blocks can improve the accuracy of stereo matching (SM), which subsequently enhances the resulting VS. To overcome the problem of border misalignment in VS, a novel edge alignment technique is presented for detecting and aligning misaligned borders. In addition, an unreliable region repairing (URR) technique was adopted to enhance the accuracy of DMG operations, thereby optimizing the image quality. The proposed SALC technique was verified using $$C$$C/Matlab simulations. Through the presented complexity reduction without any hardware acceleration, this design can achieve a high performance of 48 fps for synthesizing the virtual views of the CIF format (352 $$\times $$× 288) based on a portable platform equipped with a 1 GHz ARM Cortex-A9 RISC processor only. The experimental results revealed that the proposed SALC technique saves up to 96 % computational complexity for VS, and the computation time of DMG operations is reduced to approximately 2 % of that required for conventional SM. The average improved image quality was 32.46 dB (i.e., 7.33 % improvement from the original 30.46 dB) when using the presented edge alignment technique. When considering the overall efficiency evaluation, the proposed SALC-based SM method is 172 times more efficient than the conventional fixed block-sized SM based on the combined indices of the BPCT product. Compared with the designs of previous studies, the proposed SALC-based design obtained superior efficiency in both VS and DMG as a result of the computation reuse property.

[1]  Gauthier Lafruit,et al.  Cross-Based Local Stereo Matching Using Orthogonal Integral Images , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[2]  T. Fujii,et al.  View Generation by Ray-Space Method in Circular Camera Setup for FTV , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[3]  Liang-Gee Chen,et al.  Cache-based integer motion/disparity estimation for quad-HD H.264/AVC and HD multiview video coding , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[4]  Toshiaki Fujii,et al.  Error supression in view synthesis using reliability reasoning for FTV , 2010, 2010 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[5]  Tian-Sheuan Chang,et al.  Data reuse analysis of local stereo matching , 2008, 2008 IEEE International Symposium on Circuits and Systems.

[6]  Toshiaki Fujii,et al.  View Generation with 3D Warping Using Depth Information for FTV , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[7]  Rama Chellappa,et al.  Video synthesis of arbitrary views for approximately planar scenes , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[8]  Gauthier Lafruit,et al.  Stream-Centric Stereo Matching and View Synthesis: A High-Speed Approach on GPUs , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[9]  James J. Little,et al.  Optimizing Multiple Object Tracking and Best View Video Synthesis , 2008, IEEE Transactions on Multimedia.

[10]  Jeremy R. Cooperstock,et al.  High-Resolution Video Synthesis from Mixed-Resolution Video Based on the Estimate-and-Correct Method , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[11]  Toshiaki Fujii,et al.  Multiview Video Coding Using View Interpolation and Color Correction , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[12]  Tian-Sheuan Chang,et al.  Algorithm and Architecture of Disparity Estimation With Mini-Census Adaptive Support Weight , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[13]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Federico Tombari,et al.  Segmentation-Based Adaptive Support for Accurate Stereo Correspondence , 2007, PSIVT.

[15]  Ruigang Yang,et al.  A Performance Study on Different Cost Aggregation Approaches Used in Real-Time Stereo Matching , 2007, International Journal of Computer Vision.

[16]  Hore,et al.  [IEEE 2010 20th International Conference on Pattern Recognition (ICPR) - Istanbul, Turkey (2010.08.23-2010.08.26)] 2010 20th International Conference on Pattern Recognition - Image Quality Metrics: PSNR vs. SSIM , 2010 .

[17]  Chein-Wei Jen,et al.  A simple processor core design for DCT/IDCT , 2000, IEEE Trans. Circuits Syst. Video Technol..

[18]  H. Saito,et al.  3DTV View Generation Using Uncalibrated Cameras , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[19]  Hong Jeong,et al.  Intermediate View Synthesis for Multi-view 3D Displays Using Belief Propagation-Based Stereo Matching , 2008, 2008 Third International Conference on Convergence and Hybrid Information Technology.

[20]  Chein-Wei Jen,et al.  Hardware-efficient DFT designs with cyclic convolution and subexpression sharing , 2000 .

[21]  Yong Huang,et al.  Texture decomposition by harmonics extraction from higher order statistics , 2004, IEEE Trans. Image Process..

[22]  A. Amanatiadis,et al.  Dense disparity estimation using a hierarchical matching technique from uncalibrated stereo vision , 2009, 2009 IEEE International Workshop on Imaging Systems and Techniques.

[23]  Masanori Hariyama,et al.  Architecture of a stereo matching VLSI processor based on hierarchically parallel memory access , 2004 .

[24]  Tian-Sheuan Chang,et al.  Fast stereo matching with predictive search range , 2010, 28th Picture Coding Symposium.

[25]  Takeo Kanade,et al.  Appearance-based virtual view generation from multicamera videos captured in the 3-D room , 2003, IEEE Trans. Multim..

[26]  Jie Liang,et al.  Projective rectification-based view interpolation for multiview video coding and free viewpoint generation , 2009, 2009 Picture Coding Symposium.

[27]  Davide Aliprandi,et al.  Image-based three-dimensional free viewpoint video synthesis , 2009, 2009 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[28]  Aljoscha Smolic,et al.  Reliability-based generation and view synthesis in layered depth video , 2008, 2008 IEEE 10th Workshop on Multimedia Signal Processing.

[29]  Djemel Ziou,et al.  Image Quality Metrics: PSNR vs. SSIM , 2010, 2010 20th International Conference on Pattern Recognition.

[30]  Hideo Saito,et al.  Free viewpoint video synthesis and presentation from multiple sporting videos , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[31]  Aljoscha Smolic,et al.  The effects of multiview depth video compression on multiview rendering , 2009, Signal Process. Image Commun..

[32]  Tian-Sheuan Chang,et al.  Low-memory cost belief propagation architecture for disparity estimation , 2009, 2009 IEEE International Symposium on Circuits and Systems.

[33]  Zhang Ming,et al.  Depth-image based view synthesis for threedimensional television , 2009, 2009 4th IEEE Conference on Industrial Electronics and Applications.

[34]  Wen-Nung Lie,et al.  Intermediate view synthesis from binocular images for stereoscopic applications , 2001, ISCAS 2001. The 2001 IEEE International Symposium on Circuits and Systems (Cat. No.01CH37196).

[35]  Thomas Sikora,et al.  Adaptive Image Warping for Hole Prevention in 3D View Synthesis , 2013, IEEE Transactions on Image Processing.

[36]  Tian-Sheuan Chang,et al.  VLSI Architecture for Real-Time HD1080p View Synthesis Engine , 2011, IEEE Transactions on Circuits and Systems for Video Technology.

[37]  Zhang Zhao-yang,et al.  Arbitrary view generation based on DIBR , 2007, 2007 International Symposium on Intelligent Signal Processing and Communication Systems.

[38]  Shang-Hong Lai,et al.  Efficient multiple virtual view generation based on reduced depth stereo image for advanced autostereoscopic displays , 2009, 2009 IEEE International Conference on Multimedia and Expo.

[39]  Liang-Gee Chen,et al.  Single iteration view interpolation for multiview video applications , 2009, 2009 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[40]  S.A. White,et al.  Applications of distributed arithmetic to digital signal processing: a tutorial review , 1989, IEEE ASSP Magazine.

[41]  Jong-Il Park,et al.  Arbitrary view generation from multiple cameras , 1997, Proceedings of International Conference on Image Processing.

[42]  D. Donoho.  Wedgelets: nearly minimax estimation of edges , 1999 .

[43]  Aljoscha Smolic,et al.  Intermediate view interpolation based on multiview video plus depth for advanced 3D video systems , 2008, 2008 15th IEEE International Conference on Image Processing.

[44]  Masanori Hariyama,et al.  FPGA implementation of a stereo matching processor based on window-parallel-and-pixel-parallel architecture , 2005 .

[45]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[46]  Aljoscha Smolic,et al.  An overview of available and emerging 3D video formats and depth enhanced stereo as efficient generic solution , 2009, 2009 Picture Coding Symposium.

[47]  Nanning Zheng,et al.  Stereo Matching Using Belief Propagation , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[48]  Federico Tombari,et al.  Classification and evaluation of cost aggregation methods for stereo correspondence , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[49]  Hideo Saito,et al.  Online video synthesis for removing occluding objects using multiple uncalibrated cameras via plane sweep algorithm , 2009, 2009 Third ACM/IEEE International Conference on Distributed Smart Cameras (ICDSC).

[50]  Sehoon Yea,et al.  View Synthesis Prediction for Rate-Overhead Reduction in FTV , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[51]  Hujun Bao,et al.  Stereoscopic Video Synthesis from a Monocular Video , 2007, IEEE Transactions on Visualization and Computer Graphics.

[52]  Ghassan Al-Regib,et al.  Hierarchical Hole-Filling For Depth-Based View Synthesis in FTV and 3D Video , 2012, IEEE Journal of Selected Topics in Signal Processing.

[53]  Kuan-Hung Chen,et al.  Choose your own viewpoint: A high-quality/low-complexity free-viewpoint 3D visual system , 2012, 2012 IEEE International Conference on Emerging Signal Processing Applications.

[54]  Sehoon Yea,et al.  RD-Optimized View Synthesis Prediction for Multiview Video Coding , 2007, 2007 IEEE International Conference on Image Processing.

[55]  Jiun-In Guo,et al.  An Energy-Aware IP Core Design for the Variable-Length DCT/IDCT Targeting at MPEG4 Shape-Adaptive Transforms , 2005, IEEE Trans. Circuits Syst. Video Technol..

[56]  Robert D. Nowak,et al.  Platelets: a multiscale approach for recovering edges and surfaces in photon-limited medical imaging , 2003, IEEE Transactions on Medical Imaging.

[57]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[58]  Hsueh-Ming Hang,et al.  New view generation from a video sequence , 1998, ISCAS '98. Proceedings of the 1998 IEEE International Symposium on Circuits and Systems (Cat. No.98CH36187).

[59]  Toshiaki Fujii,et al.  View generation with 3D warping using depth information for FTV , 2009, Signal Process. Image Commun..

[60]  Kiyoharu Aizawa,et al.  Virtual view generation by linear processing of two differently focused images , 2002, Object recognition supported by user interaction for service robots.

[61]  Hideo Saito,et al.  Virtual Viewpoint Replay for a Soccer Match by View Interpolation From Multiple Cameras , 2007, IEEE Transactions on Multimedia.