Robustifying Algorithms of Learning Latent Trees with Vector Variables

We consider learning the structures of Gaussian latent tree models with vector observations when a subset of them are arbitrarily corrupted. First, we present the sample complexities of Recursive Grouping (RG) and Chow-Liu Recursive Grouping (CLRG) without the assumption that the effective depth is bounded in the number of observed nodes, significantly generalizing the results in Choi et al. (2011). We show that Chow-Liu initialization in CLRG greatly reduces the sample complexity of RG from being exponential in the diameter of the tree to only logarithmic in the diameter for the hidden Markov model (HMM). Second, we robustify RG, CLRG, Neighbor Joining (NJ) and Spectral NJ (SNJ) by using the truncated inner product. These robustified algorithms can tolerate a number of corruptions up to the square root of the number of clean samples. Finally, we derive the first known instance-dependent impossibility result for structure learning of latent trees. The optimalities of the robust version of CLRG and NJ are verified by comparing their sample complexities and the impossibility result.

[1]  Tae-Kyun Kim,et al.  Latent Regression Forest: Structured Estimation of 3D Articulated Hand Posture , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Vincent Y. F. Tan,et al.  Learning Latent Tree Graphical Models , 2010, J. Mach. Learn. Res..

[3]  Vincent Y. F. Tan,et al.  Learning Gaussian Tree Models: Analysis of Error Exponents and Extremal Structures , 2009, IEEE Transactions on Signal Processing.

[4]  Le Song,et al.  A Spectral Algorithm for Latent Tree Graphical Models , 2011, ICML.

[5]  Brendan T. O'Connor,et al.  A Latent Variable Model for Geographic Lexical Variation , 2010, EMNLP.

[6]  Martin J. Wainwright,et al.  Information-theoretic bounds on model selection for Gaussian Markov random fields , 2010, 2010 IEEE International Symposium on Information Theory.

[7]  Le Song,et al.  Spectral Methods for Learning Multivariate Latent Tree Structure , 2011, NIPS.

[8]  Constantine Caramanis,et al.  Robust estimation of tree structured Gaussian Graphical Model , 2019, ICML.

[9]  Seth Pettie,et al.  An optimal minimum spanning tree algorithm , 2000, JACM.

[10]  Lingxiao Wang,et al.  Robust Gaussian Graphical Model Estimation with Arbitrary Corruption , 2017, ICML.

[11]  M. Marcus,et al.  An extension of the Minkowski determinant theorem , 1971 .

[12]  Yuval Kluger,et al.  Spectral neighbor joining for reconstruction of latent tree models , 2020, ArXiv.

[13]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[14]  Piotr Zwiernik,et al.  Robust estimation of tree structured models , 2021, ArXiv.

[15]  Roman Vershynin,et al.  Introduction to the non-asymptotic analysis of random matrices , 2010, Compressed Sensing.

[16]  C. N. Liu,et al.  Approximating discrete probability distributions with dependence trees , 1968, IEEE Trans. Inf. Theory.

[17]  G. Stewart.  Perturbation theory for the singular value decomposition , 1990 .

[18]  D. Robinson,et al.  Comparison of phylogenetic trees , 1981 .

[19]  Kevin Atteson,et al.  The Performance of Neighbor-Joining Methods of Phylogenetic Reconstruction , 1999, Algorithmica.

[20]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[21]  N. Saitou,et al.  The neighbor-joining method: a new method for reconstructing phylogenetic trees. , 1987, Molecular biology and evolution.

[22]  Vincent Y. F. Tan,et al.  SGA: A Robust Algorithm for Partial Recovery of Tree-Structured Graphical Models with Noisy Samples , 2021, ICML.

[23]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[24]  Shie Mannor,et al.  Robust High Dimensional Sparse Regression and Matching Pursuit , 2013, ArXiv.

[25]  Jimeng Sun,et al.  Guaranteed Scalable Learning of Latent Tree Models , 2014, UAI.

[26]  Michael I. Jordan.  Graphical Models , 2003 .

[27]  Anand D. Sarwate,et al.  Learning Tree Structures from Noisy Data , 2019, AISTATS.