D-MFVI: Distributed Mean Field Variational Inference using Bregman ADMM

Bayesian models provide a framework for probabilistic modelling of complex datasets. However, many of such models are computationally demanding especially in the presence of large datasets. On the other hand, in sensor network applications, statistical (Bayesian) parameter estimation usually needs distributed algorithms, in which both data and computation are distributed across the nodes of the network. In this paper we propose a general framework for distributed Bayesian learning using Bregman Alternating Direction Method of Multipliers (B-ADMM). We demonstrate the utility of our framework, with Mean Field Variational Bayes (MFVB) as the primitive for distributed Matrix Factorization (MF) and distributed affine structure from motion (SfM).

[1]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[2]  Hagai Attias,et al.  A Variational Bayesian Framework for Graphical Models , 1999 .

[3]  Richard J. Radke.  A Survey of Distributed Computer Vision Algorithms , 2010, Handbook of Ambient Intelligence and Smart Environments.

[4]  Pietro Perona,et al.  Evaluation of Features Detectors and Descriptors based on 3D Objects , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[5]  Qing Ling,et al.  Decentralized learning for wireless communications and networking , 2015, ArXiv.

[6]  Vladimir Pavlovic,et al.  Distributed Probabilistic Learning for Camera Networks with Missing Data , 2012, NIPS.

[7]  Juan Carlos Augusto,et al.  Handbook of Ambient Intelligence and Smart Environments , 2009 .

[8]  Roberto Tron RenVidal.  A Benchmark for the Comparison of 3-D Motion Segmentation Algorithms , 2007 .

[9]  Ruslan Salakhutdinov,et al.  Bayesian probabilistic matrix factorization using Markov chain Monte Carlo , 2008, ICML '08.

[10]  Arindam Banerjee,et al.  Bregman Alternating Direction Method of Multipliers , 2013, NIPS.

[11]  René Vidal,et al.  Distributed Computer Vision Algorithms , 2011, IEEE Signal Processing Magazine.

[12]  Ruslan Salakhutdinov,et al.  Probabilistic Matrix Factorization , 2007, NIPS.

[13]  Andre Wibisono,et al.  Streaming Variational Bayes , 2013, NIPS.

[14]  David M. Blei,et al.  Nonparametric variational inference , 2012, ICML.

[15]  Lawrence Carin,et al.  Nonparametric factor analysis with beta process priors , 2009, ICML '09.

[16]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[17]  Yee Whye Teh,et al.  A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation , 2006, NIPS.

[18]  Yehuda Koren,et al.  The BellKor solution to the Netflix Prize , 2007 .

[19]  Inderjit S. Dhillon,et al.  Information-theoretic metric learning , 2006, ICML '07.

[20]  Chong Wang,et al.  Stochastic variational inference , 2012, J. Mach. Learn. Res..

[21]  Takeo Kanade,et al.  Shape and motion from image streams under orthography: a factorization method , 1992, International Journal of Computer Vision.

[22]  Pietro Perona,et al.  Evaluation of Features Detectors and Descriptors Based on 3D Objects , 2005, ICCV.

[23]  Tapani Raiko,et al.  Tkk Reports in Information and Computer Science Practical Approaches to Principal Component Analysis in the Presence of Missing Values Tkk Reports in Information and Computer Science Practical Approaches to Principal Component Analysis in the Presence of Missing Values , 2022 .

[24]  Yee Whye Teh,et al.  Collapsed Variational Inference for HDP , 2007, NIPS.

[25]  Zoubin Ghahramani,et al.  Variational Inference for Bayesian Mixtures of Factor Analysers , 1999, NIPS.

[26]  Shlomo Zilberstein,et al.  Anytime Sensing Planning and Action: A Practical Model for Robot Control , 1993, IJCAI.

[27]  Guido Sanguinetti,et al.  Advances in Neural Information Processing Systems 24 , 2011 .

[28]  Michael I. Jordan,et al.  A Sticky HDP-HMM With Application to Speaker Diarization , 2009, 0905.2592.