Learning Non-Linear Functions With Factor Graphs

We show how to use a discrete-variable factor graph for learning non-linear continuous functions from examples. The paper proposes a scheme for embedding soft quantization in a probabilistic Bayesian graph. The quantized input variables are grouped into a compound variable that is mapped through a stochastic matrix into the discrete output distribution. Specific output values are then obtained through a process of de-quantization. The information flow carried by message propagation is bi-directional and an algorithm for learning the factor graph parameters is explicitly derived. The model, that can easily merge discrete and continuous variables, is demonstrated with examples and simulations.

[1]  Francesco Palmieri,et al.  Probability Learning and Soft Quantization in Bayesian Factor Graphs , 2012, WIRN.

[2]  David Heckerman,et al.  A Tutorial on Learning with Bayesian Networks , 1999, Innovations in Bayesian Networks.

[3]  Brendan J. Frey,et al.  Factor graphs and the sum-product algorithm , 2001, IEEE Trans. Inf. Theory.

[4]  H.-A. Loeliger,et al.  An introduction to factor graphs , 2004, IEEE Signal Process. Mag..

[5]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[6]  Francesco Palmieri,et al.  Entropic priors for short-term stochastic process classification , 2011, 14th International Conference on Information Fusion.

[7]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[8]  Andrew W. Eckford,et al.  Expectation Maximization as Message Passing - Part I: Principles and Gaussian Messages , 2009, ArXiv.

[9]  M. Tribus,et al.  Probability theory: the logic of science , 2003 .

[10]  Li Ping,et al.  The Factor Graph Approach to Model-Based Signal Processing , 2007, Proceedings of the IEEE.

[11]  Xindong Wu,et al.  A new descriptive clustering algorithm based on Nonnegative Matrix Factorization , 2008, 2008 IEEE International Conference on Granular Computing.

[12]  P. Dooren,et al.  Non-negative matrix factorization with fixed row and column sums , 2008 .

[13]  Francesco Palmieri,et al.  Objective priors from maximum entropy in data classification , 2013, Inf. Fusion.

[14]  G. Forney,et al.  Codes on graphs: normal realizations , 2000, 2000 IEEE International Symposium on Information Theory (Cat. No.00CH37060).

[15]  F. Palmieri,et al.  Entropic priors for hidden-Markov model classification , 2011, 2011 IEEE Statistical Signal Processing Workshop (SSP).

[16]  Justin Dauwels,et al.  Particle Methods as Message Passing , 2006, 2006 IEEE International Symposium on Information Theory.

[17]  Simon Haykin,et al.  Adaptive Signal Processing: Next Generation Solutions , 2010 .

[18]  Igor Vajda,et al.  Entropy expressions for multivariate continuous distributions , 2000, IEEE Trans. Inf. Theory.