论文信息 - Optimal model inference for Bayesian mixture of experts

Optimal model inference for Bayesian mixture of experts

We present an algorithm for inferring the parameter and model structure of a mixture of experts model (MoE) based on the variational Bayesian (VB) framework. First in the VB framework, we show that the model parameter and structure of a MoE can be simultaneously optimized by maximizing an objective function derived in this paper. Next, we present a deterministic algorithm to find the optimal number of experts of an MoE while avoiding local maxima. Our experimental results demonstrate the practical usefulness of the method.

Zoubin Ghahramani | N. Ueda

[1] Geoffrey E. Hinton,et al. An Alternative Model for Mixtures of Experts , 1994, NIPS.

[2] Hagai Attias,et al. Inferring Parameters and Structure of Latent Variable Models by Variational Bayes , 1999, UAI.

[3] Michael I. Jordan,et al. Learning Graphical Models with Mercer Kernels , 2002, NIPS.

[4] Geoffrey E. Hinton,et al. SMEM Algorithm for Mixture Models , 1998, Neural Computation.

[5] Geoffrey E. Hinton,et al. Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[6] H. Attias,et al. Learning parameters and structure of latent variable models by variation Bayes , 1999 .

[7] David J. C. MacKay,et al. Bayesian Interpolation , 1992, Neural Computation.

[8] Robert A. Jacobs,et al. Hierarchical Mixtures of Experts and the EM Algorithm , 1993, Neural Computation.

[9] Geoffrey E. Hinton,et al. The delve manual , 1996 .

[10] Dani Gamerman,et al. Markov Chain Monte Carlo: Stochastic Simulation for Bayesian Inference , 1997 .

[11] Steve R. Waterhouse,et al. Bayesian Methods for Mixtures of Experts , 1995, NIPS.