Approximate algorithms for neural-Bayesian approaches

We describe two specific examples of neural-Bayesian approaches for complex modeling tasks: survival analysis and multitask learning. In both cases, we can come up with reasonable priors on the parameters of the neural network. As a result, the Bayesian approaches improve their (maximum likelihood) frequentist counterparts dramatically. By illustrating their application on the models under study, we review and compare algorithms that can be used for Bayesian inference: Laplace approximation, variational algorithms, Monte Carlo sampling, and empirical Bayes.

[1]  Jonathan Baxter,et al.  A Bayesian/Information Theoretic Model of Learning to Learn via Multiple Task Sampling , 1997, Machine Learning.

[2]  H. Kappen,et al.  Neural network analysis to predict treatment outcome , 1993 .

[3]  David Mackay,et al.  Probable networks and plausible predictions - a review of practical Bayesian methods for supervised neural networks , 1995 .

[4]  Michael I. Jordan.  Learning in Graphical Models , 1999, NATO ASI Series.

[5]  J. Berger,et al.  Testing Precise Hypotheses , 1987 .

[6]  David Barber,et al.  Radial Basis Functions: A Bayesian Treatment , 1997, NIPS.

[7]  Rich Caruana,et al.  Multitask Learning , 1998, Encyclopedia of Machine Learning and Data Mining.

[8]  Geoffrey E. Hinton,et al.  A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.

[9]  C. Robert.  The Bayesian choice : a decision-theoretic motivation , 1996 .

[10]  Geoffrey E. Hinton,et al.  Keeping the neural networks simple by minimizing the description length of the weights , 1993, COLT '93.

[11]  Tom Heskes,et al.  A neural-Bayesian approach to survival analysis , 1999 .

[12]  A. F. Smith,et al.  Ridge-Type Estimators for Regression Analysis , 1974 .

[13]  Tom Heskes,et al.  Empirical Bayes for Learning to Learn , 2000, ICML.

[14]  David Barber,et al.  Ensemble Learning for Multi-Layer Networks , 1997, NIPS.

[15]  Lorien Y. Pratt,et al.  A Survey of Transfer Between Connectionist Networks , 1996, Connect. Sci..

[16]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[17]  David J. C. MacKay,et al.  Comparison of Approximate Methods for Handling Hyperparameters , 1999, Neural Computation.