Accelerating Real-Time, High-Resolution Depth Upsampling on FPGAs

While the popularity of high-resolution, computer-vision applications (e.g. mixed reality, autonomous vehicles) is increasing, there have been complementary advances in time-of-flight depth sensor resolution and quality. These advances in time-of-flight sensors provide a platform for new research into real-time, depth-upsampling algorithms targeted at high-resolution video systems with low-latency requirements. This paper describes a case study in which a previously developed bilateral-filter-style upsampling algorithm is profiled, parallelized, and accelerated on an FPGA using high-level synthesis tools from Xilinx. We show that our accelerated algorithm can effectively upsample the resolution and reduce the noise of time-of-flight sensors. We also demonstrate that this algorithm exceeds the real-time requirements of 90 frames per second necessitated by mixed-reality hardware, achieving a lower-bound speedup of 40 times over the fastest CPU-only version.

[1]  Lin Gao,et al.  Temporal Upsampling of Depth Maps Using a Hybrid Camera , 2017, IEEE Transactions on Visualization and Computer Graphics.

[2]  Radu Horaud,et al.  High-resolution depth maps based on TOF-stereo fusion , 2012, 2012 IEEE International Conference on Robotics and Automation.

[3]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[4]  Dani Lischinski,et al.  Joint bilateral upsampling , 2007, SIGGRAPH 2007.

[5]  Richard Szeliski,et al.  High-accuracy stereo depth maps using structured light , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[6]  Ryan Shea,et al.  Towards Fully Offloaded Cloud-based AR: Design, Implementation and Experience , 2017, MMSys.

[7]  Christopher Joseph Pal,et al.  Learning Conditional Random Fields for Stereo , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Radu Horaud,et al.  Fusion of Range and Stereo Data for High-Resolution Scene-Modeling , 2015, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Matthias Kuba,et al.  An FPGA-Based Fully Synchronized Design of a Bilateral Filter for Real-Time Image Denoising , 2014, IEEE Transactions on Industrial Electronics.

[10]  Dmitry Chetverikov,et al.  Image-guided ToF depth upsampling: a survey , 2017, Machine Vision and Applications.

[11]  J. M. Bull,et al.  Measuring Synchronisation and Scheduling Overheads in OpenMP , 2007 .

[12]  Sebastian Thrun,et al.  A Noise‐aware Filter for Real‐time Depth Upsampling , 2008 .

[13]  Xin Jin,et al.  Depth map super-resolution via low-resolution depth guided joint trilateral up-sampling , 2017, J. Vis. Commun. Image Represent..