A Non-commutative Extension of Lee-Seung's Algorithm for Positive Semidefinite Factorizations

Given a data matrixX ∈ Rm×n + with non-negative entries, a Positive Semidefinite (PSD) factorization of X is a collection of r × r-dimensional PSD matrices {Ai} and {Bj} satisfying the condition Xij = tr(AiBj) for all i ∈ [m], j ∈ [n]. PSD factorizations are fundamentally linked to understanding the expressiveness of semidefinite programs as well as the power and limitations of quantum resources in information theory. The PSD factorization task generalizes the Nonnegative Matrix Factorization (NMF) problem in which we seek a collection of r-dimensional non-negative vectors {ai} and {bj} satisfying Xij = ai bj , for all i ∈ [m], j ∈ [n] – one can recover the latter problem by choosing matrices in the PSD factorization to be diagonal. The most widely used algorithm for computing NMFs of a matrix is the Multiplicative Update algorithm developed by Lee and Seung, in which non-negativity of the updates is preserved by scaling with positive diagonal matrices. In this paper, we describe a non-commutative extension of Lee-Seung’s algorithm, which we call the Matrix Multiplicative Update (MMU) algorithm, for computing PSD factorizations. The MMU algorithm ensures that updates remain PSD by congruence scaling with the matrix geometric mean of appropriate PSD matrices, and it retains the simplicity of implementation that the multiplicative update algorithm for NMF enjoys. Building on the MajorizationMinimization framework, we show that under our update scheme the squared loss objective is non-increasing and fixed points correspond to critical points. The analysis relies on Lieb’s Concavity Theorem. Beyond PSD factorizations, we show that the MMU algorithm can be also used as a primitive to calculate blockdiagonal PSD factorizations and tensor PSD factorizations. We demonstrate the utility of our method with experiments on real and synthetic data. 35th Conference on Neural Information Processing Systems (NeurIPS 2021), Sydney, Australia.

[1]  Nicolas Gillis,et al.  Algorithms for positive semidefinite factorization , 2017, Comput. Optim. Appl..

[2]  Aditi Raghunathan,et al.  Semidefinite relaxations for certifying robustness to adversarial examples , 2018, NeurIPS.

[3]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[4]  Stan Z. Li,et al.  Learning spatially localized, parts-based representation , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[5]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[6]  Nancy Bertin,et al.  Nonnegative Matrix Factorization with the Itakura-Saito Divergence: With Application to Music Analysis , 2009, Neural Computation.

[7]  Nicolas Gillis,et al.  Accelerated Multiplicative Updates and Hierarchical ALS Algorithms for Nonnegative Matrix Factorization , 2011, Neural Computation.

[8]  Rahul Jain,et al.  Multipartite Quantum Correlation and Communication Complexities , 2014, computational complexity.

[9]  Dana Lahat,et al.  Positive Semidefinite Matrix Factorization Based on Truncated Wirtinger Flow , 2021, 2020 28th European Signal Processing Conference (EUSIPCO).

[10]  Kim-Chuan Toh,et al.  An inexact primal–dual path following algorithm for convex quadratic SDP , 2007, Math. Program..

[11]  Inderjit S. Dhillon,et al.  Guaranteed Rank Minimization via Singular Value Projection , 2009, NIPS.

[12]  Cédric Févotte,et al.  Positive Semidefinite Matrix Factorization: A Link to Phase Retrieval And A Block Gradient Algorithm , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[13]  N. Higham.  COMPUTING A NEAREST SYMMETRIC POSITIVE SEMIDEFINITE MATRIX , 1988 .

[14]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[15]  S. Low,et al.  Zero Duality Gap in Optimal Power Flow Problem , 2012, IEEE Transactions on Power Systems.

[16]  Jakub Marecek,et al.  A low-rank coordinate-descent algorithm for semidefinite programming relaxations of optimal power flow , 2015, Optim. Methods Softw..

[17]  Max Simchowitz,et al.  Low-rank Solutions of Linear Matrix Equations via Procrustes Flow , 2015, ICML.

[18]  Mikkel N. Schmidt,et al.  Single-channel speech separation using sparse non-negative matrix factorization , 2006, INTERSPEECH.

[19]  Pablo A. Parrilo,et al.  Exponential lower bounds on fixed-size psd rank and semidefinite extension complexity , 2013, ArXiv.

[20]  Andrea Montanari,et al.  Inference in Graphical Models via Semidefinite Programming Hierarchies , 2017, NIPS.

[21]  Zhaohui Wei,et al.  Quantum and Classical Hybrid Generations for Classical Correlations , 2022, IEEE Transactions on Information Theory.

[22]  Yin Zhang,et al.  Accelerating the Lee-Seung Algorithm for Nonnegative Matrix Factorization , 2005 .

[23]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[24]  Renato D. C. Monteiro,et al.  A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization , 2003, Math. Program..

[25]  Rekha R. Thomas,et al.  Positive semidefinite rank , 2014, Math. Program..

[26]  Vincent Y. F. Tan,et al.  Positive Semidefinite Matrix Factorization: A Connection With Phase Retrieval and Affine Rank Minimization , 2021, IEEE Transactions on Signal Processing.

[27]  Rahul Jain,et al.  Efficient Protocols for Generating Bipartite Classical Distributions and Quantum States , 2013, IEEE Transactions on Information Theory.

[28]  Hans Raj Tiwary,et al.  Exponential Lower Bounds for Polytopes in Combinatorial Optimization , 2011, J. ACM.

[29]  Yuan Gao,et al.  Improving molecular cancer class discovery through sparse non-negative matrix factorization , 2005 .

[30]  Nicolas Gillis,et al.  The Why and How of Nonnegative Matrix Factorization , 2014, ArXiv.

[31]  Mihalis Yannakakis,et al.  Expressing combinatorial optimization problems by linear programs , 1991, STOC '88.

[32]  R. Bhatia.  Positive Definite Matrices , 2007 .

[33]  Rekha R. Thomas,et al.  Lifts of Convex Sets and Cone Factorizations , 2011, Math. Oper. Res..

[34]  G. Papavassilopoulos,et al.  On the rank minimization problem over a positive semidefinite linear matrix inequality , 1997, IEEE Trans. Autom. Control..

[35]  E. Carlen.  TRACE INEQUALITIES AND QUANTUM ENTROPY: An introductory course , 2009 .

[36]  Xin Liu,et al.  Document clustering based on non-negative matrix factorization , 2003, SIGIR.

[37]  Maryam Fazel,et al.  New Restricted Isometry results for noisy low-rank recovery , 2010, 2010 IEEE International Symposium on Information Theory.