Probabilistic Graphical Models and Tensor Networks: A Hybrid Framework

We investigate a correspondence between two formalisms for discrete probabilistic modeling: probabilistic graphical models (PGMs) and tensor networks (TNs), a powerful modeling framework for simulating complex quantum systems. The graphical calculus of PGMs and TNs exhibits many similarities, with discrete undirected graphical models (UGMs) being a special case of TNs. However, more general probabilistic TN models such as Born machines (BMs) employ complex-valued hidden states to produce novel forms of correlation among the probabilities. While representing a new modeling resource for capturing structure in discrete probability distributions, this behavior also renders the direct application of standard PGM tools impossible. We aim to bridge this gap by introducing a hybrid PGM-TN formalism that integrates quantum-like correlations into PGM models in a principled manner, using the physically-motivated concept of decoherence. We first prove that applying decoherence to the entirety of a BM model converts it into a discrete UGM, and conversely, that any subgraph of a discrete UGM can be represented as a decohered BM. This method allows a broad family of probabilistic TN models to be encoded as partially decohered BMs, a fact we leverage to combine the representational strengths of both model families. We experimentally verify the performance of such hybrid models in a sequential modeling task, and identify promising uses of our method within the context of existing applications of graphical models.

[1]  Alex Wang,et al.  BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model , 2019, Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation.

[2]  Yiannis Vlassopoulos,et al.  Tensor network language model , 2017, ArXiv.

[3]  Andrzej Cichocki,et al.  Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 1 Low-Rank Tensor Decompositions , 2016, Found. Trends Mach. Learn..

[4]  Alexander Novikov,et al.  Exponential Machines , 2017, ICLR.

[5]  Zhengyu Zhang,et al.  An efficient quantum algorithm for generative machine learning , 2017, ArXiv.

[6]  Nikos Komodakis,et al.  Markov Random Field modeling, inference & learning in computer vision & image understanding: A survey , 2013, Comput. Vis. Image Underst..

[7]  Alexander Novikov,et al.  Tensorizing Neural Networks , 2015, NIPS.

[8]  J. Eisert,et al.  Colloquium: Area laws for the entanglement entropy , 2010 .

[9]  D. Poulin,et al.  Quantum Graphical Models and Belief Propagation , 2007, 0708.1337.

[10]  Guillaume Rabusseau,et al.  Tensor Networks for Probabilistic Sequence Modeling , 2020, AISTATS.

[11]  Dusko Pavlovic,et al.  A new description of orthogonal bases , 2008, Mathematical Structures in Computer Science.

[12]  Herbert Jaeger,et al.  Norm-Observable Operator Models , 2010, Neural Computation.

[13]  Amnon Shashua,et al.  Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Design , 2017, ICLR.

[14]  Yiannis Vlassopoulos,et al.  Language as a matrix product state , 2017, ArXiv.

[15]  W. Bialek,et al.  Are Biological Systems Poised at Criticality? , 2010, 1012.2242.

[16]  F. Verstraete,et al.  Matrix product states, projected entangled pair states, and variational renormalization group methods for quantum spin systems , 2008, 0907.2796.

[17]  J. Cirac,et al.  Tensor networks and efficient descriptions of classical data , 2021, ArXiv.

[18]  Byron Boots,et al.  Quantum Tensor Networks, Stochastic Processes, and Weighted Automata , 2020, AISTATS.

[19]  David I. Spivak,et al.  An Invitation to Applied Category Theory , 2019 .

[20]  David J. Schwab,et al.  Supervised Learning with Tensor Networks , 2016, NIPS.

[21]  Xun Gao,et al.  Enhancing Generative Models via Quantum Correlations , 2021 .

[22]  W. Zurek.  Decoherence, einselection, and the quantum origins of the classical , 2001, quant-ph/0105127.

[23]  James Stokes,et al.  Probabilistic Modeling with Matrix Product States , 2019, Entropy.

[24]  Andrew J. Ferris,et al.  Perfect Sampling with Unitary Tensor Networks , 2012, 1201.3974.

[25]  E. Miles Stoudenmire,et al.  Learning relevant features of data with multi-scale tensor networks , 2017, ArXiv.

[26]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[27]  Alejandro Perdomo-Ortiz,et al.  A generative modeling approach for benchmarking and training shallow quantum circuits , 2018, npj Quantum Information.

[28]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[29]  Elina Robeva,et al.  Duality of Graphical Models and Tensor Networks , 2017, Information and Inference: A Journal of the IMA.

[30]  Nadav Cohen,et al.  On the Expressive Power of Deep Learning: A Tensor Analysis , 2015, COLT 2016.

[31]  Jun Wang,et al.  Unsupervised Generative Modeling Using Matrix Product States , 2017, Physical Review X.

[32]  Raphaël Bailly.  Quadratic Weighted Automata: Spectral Algorithm and Likelihood Maximization , 2011, ACML 2011.

[33]  John D. Hunter,et al.  Matplotlib: A 2D Graphics Environment , 2007, Computing in Science & Engineering.

[34]  John Preskill,et al.  Quantum Computing in the NISQ era and beyond , 2018, Quantum.

[35]  Jens Eisert,et al.  Expressive power of tensor-network factorizations for probabilistic modeling, with applications from hidden Markov models to quantum machine learning , 2019, NeurIPS.

[36]  Roman Orus,et al.  A Practical Introduction to Tensor Networks: Matrix Product States and Projected Entangled Pair States , 2013, 1306.2164.

[37]  K. Jarrod Millman,et al.  Array programming with NumPy , 2020, Nat..

[38]  R. Demarr,et al.  Nonnegative matrices with nonnegative inverses , 1972 .