Highly parallel steered mixture-of-experts rendering at pixel-level for image and light field data

A novel image approximation framework called steered mixture-of-experts (SMoE) was recently presented. SMoE has multiple applications in coding, scale-conversion, and general processing of image modalities. In particular, it has strong potential for coding and streaming higher dimensional image modalities that are necessary to leverage full translational and rotational freedom (6 degrees-of-freedom) in virtual reality for camera captured images. In this paper, we analyze the rendering performance of SMoE for 2D images and 4D light fields. Two different GPU implementations that parallelize the SMoE regression step at pixel-level are presented, including experimental evaluations based on rendering performance and quality. In this paper it is shown that on appropriate hardware, an OpenCL implementation can achieve 85 fps and 22 fps for, respectively, 1080p and 4K renderings of large models with more than 100,000 of Gaussian kernels.

[1]  Peter Lambert,et al.  A universal image coding approach using sparse steered Mixture-of-Experts regression , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[2]  Philipp Birken,et al.  Numerical Linear Algebra , 2011, Encyclopedia of Parallel Computing.

[3]  Christian Böhm,et al.  Massively parallel expectation maximization using graphics processing units , 2013, KDD.

[4]  Bart Goossens,et al.  Dataflow management, dynamic load balancing, and concurrent processing for real‐time embedded vision applications using Quasar , 2018, Int. J. Circuit Theory Appl..

[5]  Touradj Ebrahimi,et al.  Objective and subjective evaluation of light field image compression algorithms , 2016, 2016 Picture Coding Symposium (PCS).

[6]  William H. Press,et al.  Numerical Recipes 3rd Edition: The Art of Scientific Computing , 2007 .

[7]  D. Broomhead,et al.  Radial Basis Functions, Multi-Variable Functional Interpolation and Adaptive Networks , 1988 .

[8]  Qionghai Dai,et al.  Light Field Image Processing: An Overview , 2017, IEEE Journal of Selected Topics in Signal Processing.

[9]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[10]  Thomas Sikora,et al.  Regularized Gradient Descent Training of Steered Mixture of Experts for Sparse Image Representation , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[11]  Peter Lambert,et al.  Progressive Modeling of Steered Mixture-of-Experts for Light Field Video Approximation , 2018, 2018 Picture Coding Symposium (PCS).

[12]  Ivo Ihrke,et al.  Principles of Light Field Imaging: Briefly revisiting 25 years of research , 2016, IEEE Signal Processing Magazine.

[13]  John E. Stone,et al.  OpenCL: A Parallel Programming Standard for Heterogeneous Computing Systems , 2010, Computing in Science & Engineering.

[14]  Nicholas Wilt,et al.  The CUDA Handbook: A Comprehensive Guide to GPU Programming , 2013 .

[15]  Guido Bugmann,et al.  Normalized Gaussian Radial Basis Function networks , 1998, Neurocomputing.

[16]  Peter Lambert,et al.  Steered mixture-of-experts for light field coding, depth estimation, and processing , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[17]  Ian Buck,et al.  Fast Parallel Expectation Maximization for Gaussian Mixture Models on GPUs Using CUDA , 2009, 2009 11th IEEE International Conference on High Performance Computing and Communications.

[18]  Vlastimil Havran,et al.  Register Efficient Dynamic Memory Allocator for GPUs , 2015, Comput. Graph. Forum.

[19]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[20]  Krzysztof Wegner,et al.  Immersive visual media — MPEG-I: 360 video, virtual navigation and beyond , 2017, 2017 International Conference on Systems, Signals and Image Processing (IWSSIP).

[21]  Wilfried Philips,et al.  Quasar — A new heterogeneous programming framework for image and video processing algorithms on CPU and GPU , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[22]  Hiroaki Kobayashi,et al.  Optimized Data Transfers Based on the OpenCL Event Management Mechanism , 2015, Sci. Program..

[23]  Pablo Carballeira,et al.  Toward the realization of six degrees-of-freedom with compressed light fields , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[24]  Thomas Sikora,et al.  Video representation and coding using a sparse steered mixture-of-experts network , 2016, 2016 Picture Coding Symposium (PCS).

[25]  Joseph N. Wilson,et al.  Twenty Years of Mixture of Experts , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[26]  E. Adelson,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[27]  Thomas Sikora,et al.  An MSE Approach For Training And Coding Steered Mixtures Of Experts , 2018, 2018 Picture Coding Symposium (PCS).

[28]  Aaftab Munshi,et al.  The OpenCL specification , 2009, 2009 IEEE Hot Chips 21 Symposium (HCS).

[29]  Charles Elkan,et al.  Expectation Maximization Algorithm , 2010, Encyclopedia of Machine Learning.

[30]  Touradj Ebrahimi,et al.  New Light Field Image Dataset , 2016, QoMEX 2016.

[31]  H. Sung.  Gaussian Mixture Regression and Classification , 2004 .

[32]  Gary J. Sullivan,et al.  Overview of the High Efficiency Video Coding (HEVC) Standard , 2012, IEEE Transactions on Circuits and Systems for Video Technology.