Variational Bayesian Methods For Multimedia Problems

In this paper we present an introduction to Variational Bayesian (VB) methods in the context of probabilistic graphical models, and discuss their application in multimedia related problems. VB is a family of deterministic probability distribution approximation procedures that offer distinct advantages over alternative approaches based on stochastic sampling and those providing only point estimates. VB inference is flexible to be applied in different practical problems, yet is broad enough to subsume as its special cases several alternative inference approaches including Maximum A Posteriori (MAP) and the Expectation-Maximization (EM) algorithm. In this paper we also show the connections between VB and other posterior approximation methods such as the marginalization-based Loopy Belief Propagation (LBP) and the Expectation Propagation (EP) algorithms. Specifically, both VB and EP are variational methods that minimize functionals based on the Kullback-Leibler (KL) divergence. LBP, traditionally developed using graphical models, can also be viewed as a VB inference procedure. We present several multimedia related applications illustrating the use and effectiveness of the VB algorithms discussed herein. We hope that by reading this tutorial the readers will obtain a general understanding of Bayesian methods and establish connections among popular algorithms used in practice.

[1]  Jin Wang,et al.  A low-rank matrix completion based intra prediction for H.264/AVC , 2011, 2011 IEEE 13th International Workshop on Multimedia Signal Processing.

[2]  John Wright,et al.  RASL: Robust alignment by sparse and low-rank decomposition for linearly correlated images , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[3]  Chong Wang,et al.  Simultaneous image classification and annotation , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  R. Rockafellar Convex Analysis: (pms-28) , 1970 .

[5]  丸山 徹 Convex Analysisの二,三の進展について , 1977 .

[6]  Farid Melgani,et al.  Gaussian Process Approach to Remote Sensing Image Classification , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[7]  Aggelos K. Katsaggelos,et al.  Sparse Bayesian Methods for Low-Rank Matrix Estimation , 2011, IEEE Transactions on Signal Processing.

[8]  Santosh S. Vempala,et al.  Latent Semantic Indexing , 2000, PODS 2000.

[9]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine Learning.

[10]  Bhaskar D. Rao,et al.  Variational EM Algorithms for Non-Gaussian Latent Variable Models , 2005, NIPS.

[11]  Simon J. D. Prince,et al.  Computer Vision: Models, Learning, and Inference , 2012 .

[12]  Massimo Piccardi,et al.  Background subtraction techniques: a review , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[13]  M. C. Smith,et al.  Parallelizing Principal Component Analysis for Robust Facial Recognition Using CUDA , 2012, 2012 Symposium on Application Accelerators in High Performance Computing.

[14]  Harri Lappalainen,et al.  Ensemble learning for independent component analysis , 1999 .

[15]  Aggelos K. Katsaggelos,et al.  A Variational Approach for Sparse Component Estimation and Low-Rank Matrix Recovery , 2013, J. Commun..

[16]  Thomas Buchgraber,et al.  Variational Sparse Bayesian Learning: Centralized and Distributed Processing , 2013 .

[17]  David M. Blei,et al.  Supervised Topic Models , 2007, NIPS.

[18]  Santosh S. Vempala,et al.  Latent semantic indexing: a probabilistic analysis , 1998, PODS '98.

[19]  David Barber,et al.  Bayesian reasoning and machine learning , 2012 .

[20]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[21]  Yi Ma,et al.  Face recovery in conference video streaming using robust principal component analysis , 2011, 2011 18th IEEE International Conference on Image Processing.

[22]  R. Horgan,et al.  Statistical Field Theory , 2014 .

[23]  Jiming Liu,et al.  Learning Topic Models by Belief Propagation , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Aggelos K. Katsaggelos,et al.  Variational Bayesian Blind Deconvolution Using a Total Variation Prior , 2009, IEEE Transactions on Image Processing.

[25]  G. Tian,et al.  Statistical Applications in Genetics and Molecular Biology Sparse Logistic Regression with Lp Penalty for Biomarker Identification , 2011 .

[26]  Michael I. Jordan,et al.  Bayesian parameter estimation via variational methods , 2000, Stat. Comput..

[27]  Radford M. Neal Pattern Recognition and Machine Learning , 2007, Technometrics.

[28]  Justin Grimmer An Introduction to Bayesian Inference via Variational Approximations , 2011, Political Analysis.

[29]  Tanja Schultz,et al.  Dynamic language model adaptation using variational Bayes inference , 2005, INTERSPEECH.

[30]  Stephen J. Roberts,et al.  A tutorial on variational Bayesian inference , 2012, Artificial Intelligence Review.

[31]  T. Tran,et al.  Video concealment via matrix completion at high missing rates , 2010, 2010 Conference Record of the Forty Fourth Asilomar Conference on Signals, Systems and Computers.

[32]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[33]  Karl J. Friston,et al.  Variational Bayesian inference for fMRI time series , 2003, NeuroImage.

[34]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Antonio Torralba,et al.  LabelMe: A Database and Web-Based Tool for Image Annotation , 2008, International Journal of Computer Vision.

[36]  Juergen Luettin,et al.  Audio-Visual Speech Modeling for Continuous Speech Recognition , 2000, IEEE Trans. Multim..

[37]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[38]  Justin P. Haldar,et al.  Spatiotemporal imaging with partially separable functions: A matrix recovery approach , 2010, 2010 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[39]  Hossein Mobahi,et al.  Toward a Practical Face Recognition System: Robust Alignment and Illumination by Sparse Representation , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[41]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[42]  Naonori Ueda,et al.  Variational bayesian estimation and clustering for speech recognition , 2004, IEEE Transactions on Speech and Audio Processing.

[43]  Yen-Wei Chen,et al.  Ensemble learning for independent component analysis , 2006, Pattern Recognit..

[44]  D.G. Tzikas,et al.  The variational approximation for Bayesian inference , 2008, IEEE Signal Processing Magazine.

[45]  Ata Kabán,et al.  Learning with Lq<1 vs L1-Norm Regularisation with Exponentially Many Irrelevant Features , 2008, ECML/PKDD.

[46]  Simon J. D. Prince,et al.  Computer Vision: Index , 2012 .

[47]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[48]  Nathan Srebro,et al.  Learning with matrix factorizations , 2004 .

[49]  Kevin P. Murphy,et al.  Dynamic Bayesian Networks for Audio-Visual Speech Recognition , 2002, EURASIP J. Adv. Signal Process..

[50]  Alan M. McIvor,et al.  Background Subtraction Techniques , 2000 .

[51]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[52]  A. Montanari,et al.  Sensor network localization from local connectivity: Performance analysis for the MDS-MAP algorithm , 2010, 2010 IEEE Information Theory Workshop on Information Theory (ITW 2010, Cairo).

[53]  Xiangyang Wang,et al.  Multi-Task low-rank and sparse matrix recovery for human motion segmentation , 2012, 2012 19th IEEE International Conference on Image Processing.

[54]  W. Eric L. Grimson,et al.  Unsupervised Activity Perception by Hierarchical Bayesian Models , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[55]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[56]  Farid Melgani,et al.  Classification of Hyperspectral Remote Sensing Images Using Gaussian Processes , 2008, IGARSS 2008 - 2008 IEEE International Geoscience and Remote Sensing Symposium.

[57]  Zuowei Shen,et al.  Robust video denoising using low rank matrix completion , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[58]  Tom Minka,et al.  Expectation Propagation for approximate Bayesian inference , 2001, UAI.

[59]  David Suter,et al.  Recovering the missing components in a large noisy low-rank matrix: application to SFM , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[60]  W. Eric L. Grimson,et al.  Unsupervised Activity Perception in Crowded and Complicated Scenes Using Hierarchical Bayesian Models , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[61]  George Eastman House,et al.  Sparse Bayesian Learning and the Relevan e Ve tor Ma hine , 2001 .

[62]  Xin Li,et al.  Patch-Based Video Processing: A Variational Bayesian Approach , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[63]  Lawrence Carin,et al.  Infinite Hidden Markov Models for Unusual-Event Detection in Video , 2008, IEEE Transactions on Image Processing.

[64]  Tom Minka,et al.  A family of algorithms for approximate Bayesian inference , 2001 .

[65]  Paris Smaragdis,et al.  Singing-voice separation from monaural recordings using robust principal component analysis , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[66]  Lieven Vandenberghe,et al.  Interior-Point Method for Nuclear Norm Approximation with Application to System Identification , 2009, SIAM J. Matrix Anal. Appl..

[67]  Jung-Min Park,et al.  An overview of anomaly detection techniques: Existing solutions and latest technological trends , 2007, Comput. Networks.

[68]  Matthew J. Beal Variational algorithms for approximate Bayesian inference , 2003 .

[69]  Pramode K. Verma,et al.  An efficient video denoising method using decomposition approach for low-rank matrix completion , 2012, 2012 Conference Record of the Forty Sixth Asilomar Conference on Signals, Systems and Computers (ASILOMAR).

[70]  Aggelos K. Katsaggelos,et al.  Bayesian Blind Deconvolution with General Sparse Image Priors , 2012, ECCV.