Alternative Variable Splitting Methods to Learn Sum-Product Networks

Sum-Product Networks (SPNs) are recent deep probabilistic models providing exact and tractable inference. SPNs have been successfully employed as density estimators in several application domains. However, learning an SPN from high dimensional data still poses a challenge in terms of time complexity. This is due to the high cost of determining independencies among random variables (RVs) and sub-populations among samples, two operations that are repeated several times. Even one of the simplest greedy structure learner, LearnSPN, scales quadratically in the number of the variables to determine RVs independencies. In this work we investigate approximate but fast procedures to determine independencies among RVs whose complexity scales in sub-quadratic time. We propose two procedures: a random subspace approach and one that adopts entropy as a criterion to split RVs in linear time. Experimental results prove that LearnSPN equipped by our splitting procedures is able to reduce learning and/or inference times while preserving comparable inference accuracy.

[1]  Franz Pernkopf,et al.  Modeling speech with sum-product networks: Application to bandwidth extension , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2]  Dan Ventura,et al.  Learning the Architecture of Sum-Product Networks Using Clustering on Variables , 2012, NIPS.

[3]  Jesse Davis,et al.  Markov Network Structure Learning: A Randomized Feature Generation Approach , 2012, AAAI.

[4]  Floriana Esposito,et al.  Simplifying, Regularizing and Strengthening Sum-Product Network Structure Learning , 2015, ECML/PKDD.

[5]  Adnan Darwiche,et al.  A differential approach to inference in Bayesian networks , 2000, JACM.

[6]  Vibhav Gogate,et al.  Cutset Networks: A Simple, Tractable, and Scalable Approach for Improving the Accuracy of Chow-Liu Trees , 2014, ECML/PKDD.

[7]  Daniel Lowd,et al.  Learning Sum-Product Networks with Direct and Indirect Variable Interactions , 2014, ICML.

[8]  Pedro M. Domingos,et al.  Learning the Structure of Sum-Product Networks , 2013, ICML.

[9]  James Martens,et al.  On the Expressive Efficiency of Sum Product Networks , 2014, ArXiv.

[10]  Floriana Esposito,et al.  Learning Accurate Cutset Networks by Exploiting Decomposability , 2015, AI*IA.

[11]  Pedro M. Domingos,et al.  Sum-product networks: A new deep architecture , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[12]  Jesse Davis,et al.  Learning Markov Network Structure with Decision Trees , 2010, 2010 IEEE International Conference on Data Mining.

[13]  Hao Wang,et al.  Modeling spatial layout for scene image understanding via a novel multiscale sum-product network , 2016, Expert Syst. Appl..

[14]  Wei-Chen Cheng,et al.  Language modeling with sum-product networks , 2014, INTERSPEECH.

[15]  Dan Roth,et al.  On the Hardness of Approximate Reasoning , 1993, IJCAI.

[16]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[17]  Pedro M. Domingos,et al.  The Sum-Product Theorem: A Foundation for Learning Tractable Models , 2016, ICML.

[18]  Floriana Esposito,et al.  Visualizing and understanding Sum-Product Networks , 2016, Machine Learning.

[19]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[20]  Kristian Kersting,et al.  Poisson Sum-Product Networks: A Deep Architecture for Tractable Multivariate Poisson Distributions , 2017, AAAI.

[21]  Maurice Queyranne,et al.  Minimizing symmetric submodular functions , 1998, Math. Program..

[22]  Teresa Maria Altomare Basile,et al.  Learning Bayesian Random Cutset Forests , 2015, ISMIS.

[23]  Ali Ghodsi,et al.  Learning the Structure of Sum-Product Networks via an SVD-based Algorithm , 2015, UAI.

[24]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.