A hardware-friendly bilateral solver for real-time virtual reality video

Rendering 3D-360° VR video from a camera rig is computation-intensive and typically performed offline. In this paper, we target the most time-consuming step of the VR video creation process, high-quality flow estimation with the bilateral solver. We propose a new algorithm, the hardware-friendly bilateral solver, that enables faster runtimes than existing algorithms of similar quality. Our algorithm is easily parallelized, achieving a 4× speedup on CPU and 32× speedup on GPU over a baseline CPU implementation. We also design an FPGA-based hardware accelerator that utilizes reduced-precision computation and the parallelism inherent in our algorithm to achieve further speedups over our CPU and GPU implementations while consuming an order of magnitude less power. The FPGA design's power efficiency enables practical real-time VR video processing at the camera rig or in the cloud.

[1]  Nathan Ickes,et al.  Reconfigurable processor for energy-scalable computational photography , 2013, 2013 IEEE International Solid-State Circuits Conference Digest of Technical Papers.

[2]  Minh N. Do,et al.  Fast Global Image Smoothing Based on Weighted Least Squares , 2014, IEEE Transactions on Image Processing.

[3]  Hans-Peter Seidel,et al.  Free-viewpoint video of human actors , 2003, ACM Trans. Graph..

[4]  Jonathan T. Barron,et al.  The Fast Bilateral Solver , 2015, ECCV.

[5]  Zeev Farbman,et al.  Edge-preserving decompositions for multi-scale tone and detail manipulation , 2008, SIGGRAPH 2008.

[6]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[7]  Sebastian Thrun,et al.  A Noise‐aware Filter for Real‐time Depth Upsampling , 2008 .

[8]  Andrew Adams,et al.  Fast High‐Dimensional Filtering Using the Permutohedral Lattice , 2010, Comput. Graph. Forum.

[9]  Jonathan T. Barron,et al.  Jump: virtual reality video , 2016, ACM Trans. Graph..

[10]  Qi Zhang,et al.  100+ Times Faster Weighted Median Filter (WMF) , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Qingxiong Yang,et al.  Stereo Matching Using Tree Filtering , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Yael Pritch,et al.  Omnistereo: Panoramic Stereo Imaging , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Harry Shum,et al.  A virtual reality system using the concentric mosaic: construction, rendering, and data compression , 2005, IEEE Trans. Multim..

[14]  Zeev Farbman,et al.  Edge-preserving decompositions for multi-scale tone and detail manipulation , 2008, ACM Trans. Graph..

[15]  Jiawen Chen,et al.  Real-time edge-aware image processing with the bilateral grid , 2007, ACM Trans. Graph..

[16]  Horst Bischof,et al.  Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation , 2013, 2013 IEEE International Conference on Computer Vision.

[17]  Jian Sun,et al.  Guided Image Filtering , 2010, ECCV.

[18]  Nathan Ickes,et al.  Reconfigurable Processor for Energy-Efficient Computational Photography , 2013, IEEE Journal of Solid-State Circuits.

[19]  Jian Sun,et al.  Fast Guided Filter , 2015, ArXiv.

[20]  Qingxiong Yang,et al.  Hardware-Efficient Bilateral Filtering for Stereo Matching , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Enhua Wu,et al.  Constant Time Weighted Median Filtering for Stereo Matching and Beyond , 2013, 2013 IEEE International Conference on Computer Vision.

[22]  Dani Lischinski,et al.  Joint bilateral upsampling , 2007, ACM Trans. Graph..

[23]  Jonathan T. Barron,et al.  Fast bilateral-space stereo for synthetic defocus , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Manuel Menezes de Oliveira Neto,et al.  Domain transform for edge-aware image and video processing , 2011, ACM Trans. Graph..

[25]  Gordon Wetzstein,et al.  The light field stereoscope , 2015, ACM Trans. Graph..

[26]  Boris Polyak.  Some methods of speeding up the convergence of iteration methods , 1964 .

[27]  Frédo Durand,et al.  Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines , 2013, PLDI 2013.

[28]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.