Streaming Model Selection via Online Factorized Asymptotic Bayesian Inference

Recent growing needs for real time data analytics have increased importance of streaming model selection. Real-world streaming observations are often obtained by dynamically-changing or heterogeneous data sources, and learning machines must identify the complexities of the data generation processes on the fly without prior knowledge. This paper proposes online FAB (OFAB) inference as a general framework for streaming model selection of latent variable models. The key idea in OFAB inference is degeneration, i.e. it intentionally considers a "redundant" latent space anddynamically derives a "non-redundant" latent sub-space using a FAB-unique shrinkage mechanism on demand. By integrating the idea of stochastic variational inference, OFAB automatically and dynamically selects the best dimensionality of latent variables in a streaming and Bayesian principled manner. Empirical results on two applications, density estimation and abnormal detection, show that online FAB (OFAB) outperformed the state-of-the-art online inference methods.

[1]  Francis R. Bach,et al.  Online Learning for Latent Dirichlet Allocation , 2010, NIPS.

[2]  Jialei Wang,et al.  Trading Interpretability for Accuracy: Oblique Treed Sparse Additive Models , 2015, KDD.

[3]  Eyke Hüllermeier,et al.  On the bayes-optimality of F-measure maximizers , 2013, J. Mach. Learn. Res..

[4]  D. Blei,et al.  Truncation-free stochastic variational inference for Bayesian nonparametric models , 2012, NIPS 2012.

[5]  Satoshi Morinaga,et al.  Factorized Asymptotic Bayesian Inference for Mixture Modeling , 2012, AISTATS.

[6]  Satoshi Morinaga,et al.  Fully-Automatic Bayesian Piecewise Sparse Linear Models , 2014, AISTATS.

[7]  Kohei Hayashi,et al.  Rebuilding Factorized Information Criterion: Asymptotically Accurate Marginal Likelihood , 2015, ICML.

[8]  Satoshi Morinaga,et al.  Online heterogeneous mixture modeling with marginal and copula selection , 2011, KDD.

[9]  Christopher M. Bishop,et al.  Mixtures of Probabilistic Principal Component Analyzers , 1999, Neural Computation.

[10]  Emily B. Fox,et al.  Streaming Variational Inference for Bayesian Nonparametric Mixture Models , 2014, AISTATS.

[11]  Andre Wibisono,et al.  Streaming Variational Bayes , 2013, NIPS.

[12]  Matthew T. Harrison,et al.  Inconsistency of Pitman-Yor process mixtures for the number of components , 2013, J. Mach. Learn. Res..

[13]  Dahua Lin,et al.  Online Learning of Nonparametric Mixture Models via Sequential Variational Approximation , 2013, NIPS.

[14]  Chong Wang,et al.  An Adaptive Learning Rate for Stochastic Variational Inference , 2013, ICML.

[15]  Yusuke Muraoka,et al.  Scalable Model Selection for Large-Scale Factorial Relational Models , 2015, ICML.

[16]  Matthew D. Hoffman,et al.  A trust-region method for stochastic variational inference with applications to streaming data , 2015, ICML.

[17]  Erik B. Sudderth,et al.  Memoized Online Variational Inference for Dirichlet Process Mixture Models , 2013, NIPS.

[18]  Chong Wang,et al.  Stochastic variational inference , 2012, J. Mach. Learn. Res..

[19]  James R. Foulds,et al.  Stochastic collapsed variational Bayesian inference for latent Dirichlet allocation , 2013, KDD.

[20]  Kohei Hayashi,et al.  Factorized Asymptotic Bayesian Hidden Markov Models , 2012, ICML.

[21]  Matthew T. Harrison,et al.  A simple example of Dirichlet process mixture inconsistency for the number of components , 2013, NIPS.

[22]  Neil D. Lawrence,et al.  Gaussian Processes for Big Data , 2013, UAI.

[23]  Kohei Hayashi,et al.  Factorized Asymptotic Bayesian Inference for Latent Feature Models , 2013, NIPS.

[24]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[25]  Erik B. Sudderth,et al.  Truly Nonparametric Online Variational Inference for Hierarchical Dirichlet Processes , 2012, NIPS.

[26]  Jonathan P. How,et al.  Streaming, Distributed Variational Inference for Bayesian Nonparametrics , 2015, NIPS.

[27]  Lawrence K. Saul,et al.  Identifying suspicious URLs: an application of large-scale online learning , 2009, ICML '09.

[28]  Chong Wang,et al.  Online Variational Inference for the Hierarchical Dirichlet Process , 2011, AISTATS.

[29]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[30]  Masa-aki Sato,et al.  Online Model Selection Based on the Variational Bayes , 2001, Neural Computation.