Active matrix completion with uncertainty quantification

The noisy matrix completion problem, which aims to recover a low-rank matrix $\mathbf{X}$ from a partial, noisy observation of its entries, arises in many statistical, machine learning, and engineering applications. In this paper, we present a new, information-theoretic approach for active sampling (or designing) of matrix entries for noisy matrix completion, based on the maximum entropy design principle. One novelty of our method is that it implicitly makes use of uncertainty quantification (UQ) -- a measure of uncertainty for unobserved matrix entries -- to guide the active sampling procedure. The proposed framework reveals several novel insights on the role of compressive sensing (e.g., coherence) and coding design (e.g., Latin squares) on the sampling performance and UQ for noisy matrix completion. Using such insights, we develop an efficient posterior sampler for UQ, which is then used to guide a closed-form sampling scheme for matrix entries. Finally, we illustrate the effectiveness of this integrated sampling / UQ methodology in simulation studies and two applications to collaborative filtering.

[1]  J. I The Design of Experiments , 1936, Nature.

[2]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[3]  E. Wigner Characteristic Vectors of Bordered Matrices with Infinite Dimensions I , 1955 .

[4]  W. K. Hastings,et al.  Monte Carlo Sampling Methods Using Markov Chains and Their Applications , 1970 .

[5]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Henry P. Wynn,et al.  Maximum entropy sampling , 1987 .

[7]  D. Rubin,et al.  Statistical Analysis with Missing Data , 1988 .

[8]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[9]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[10]  P. Matthews,et al.  Generating uniformly distributed random latin squares , 1996 .

[11]  A. Rukhin Bayes and Empirical Bayes Methods for Data Analysis , 1997 .

[12]  A. Rukhin Matrix Variate Distributions , 1999, The Multivariate Normal Distribution.

[13]  H. Wynn,et al.  Maximum entropy sampling and optimal Bayesian experimental design , 2000 .

[14]  Jianhong Shen On the singular values of Gaussian random matrices , 2001 .

[15]  Thomas J. Santner,et al.  Design and analysis of computer experiments , 1998 .

[16]  Thomas J. Santner,et al.  The Design and Analysis of Computer Experiments , 2003, Springer Series in Statistics.

[17]  Torleiv Kløve,et al.  Permutation arrays for powerline communication and mutually orthogonal latin squares , 2004, IEEE Transactions on Information Theory.

[18]  Kenneth Y. Goldberg,et al.  Eigentaste: A Constant Time Collaborative Filtering Algorithm , 2001, Information Retrieval.

[19]  Daniel Pérez Palomar,et al.  Gradient of mutual information in linear vector Gaussian channels , 2005, ISIT.

[20]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[21]  Sophie Huczynska Powerline communication and the 36 officers problem , 2006, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[22]  Stephen P. Boyd,et al.  Disciplined Convex Programming , 2006 .

[23]  Peter D. Hoff,et al.  Simulation of the Matrix Bingham–von Mises–Fisher Distribution, With Applications to Multivariate and Relational Data , 2007, 0712.4166.

[24]  Yee Whye Teh,et al.  Variational Bayesian Approach to Movie Rating Prediction , 2007, KDD 2007.

[25]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2009, Found. Comput. Math..

[26]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[27]  Neil D. Lawrence,et al.  Non-linear matrix factorization with Gaussian processes , 2009, ICML '09.

[28]  Andrea Montanari,et al.  Matrix completion from a few entries , 2009, ISIT.

[29]  Stephen Becker,et al.  Quantum state tomography via compressed sensing. , 2009, Physical review letters.

[30]  Sudhakar Prasad Certain Relations between Mutual Information and Fidelity of Statistical Estimation , 2010, ArXiv.

[31]  Emmanuel J. Candès,et al.  The Power of Convex Relaxation: Near-Optimal Matrix Completion , 2009, IEEE Transactions on Information Theory.

[32]  Emmanuel J. Candès,et al.  Matrix Completion With Noise , 2009, Proceedings of the IEEE.

[33]  Yoshua Bengio,et al.  Algorithms for Hyper-Parameter Optimization , 2011, NIPS.

[34]  Emmanuel J. Candès,et al.  Tight Oracle Inequalities for Low-Rank Matrix Recovery From a Minimal Number of Noisy Random Measurements , 2011, IEEE Transactions on Information Theory.

[35]  Aggelos K. Katsaggelos,et al.  Low-rank matrix completion by variational sparse Bayesian learning , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[36]  Benjamin Recht,et al.  A Simpler Approach to Matrix Completion , 2009, J. Mach. Learn. Res..

[37]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[38]  A. Robert Calderbank,et al.  Communications-Inspired Projection Design with Application to Compressive Sensing , 2012, SIAM J. Imaging Sci..

[39]  Martin J. Wainwright,et al.  Restricted strong convexity and weighted matrix completion: Optimal bounds with noise , 2010, J. Mach. Learn. Res..

[40]  Jiayu Zhou,et al.  Active Matrix Completion , 2013, 2013 IEEE 13th International Conference on Data Mining.

[41]  Barnabás Póczos,et al.  Active learning and search on low-rank matrices , 2013, KDD.

[42]  Nagarajan Natarajan,et al.  Inductive matrix completion for predicting gene–disease associations , 2014, Bioinform..

[43]  Ralph C. Smith,et al.  Uncertainty Quantification: Theory, Implementation, and Applications , 2013 .

[44]  Sebastian Pokutta,et al.  Info-Greedy Sequential Adaptive Compressed Sensing , 2014, IEEE Journal of Selected Topics in Signal Processing.

[45]  A. Robert Calderbank,et al.  Nonlinear Information-Theoretic Compressive Measurement Design , 2014, ICML.

[46]  Mark Crovella,et al.  Matrix Completion with Queries , 2015, KDD.

[47]  Justin K. Romberg,et al.  An Overview of Low-Rank Matrix Recovery From Incomplete Observations , 2016, IEEE Journal of Selected Topics in Signal Processing.

[48]  V. Roshan Joseph,et al.  Minimax and Minimax Projection Designs Using Clustering , 2016, 1602.03938.

[49]  Pascal Fua,et al.  Learning Active Learning from Data , 2017, NIPS.

[50]  D. Sculley,et al.  Google Vizier: A Service for Black-Box Optimization , 2017, KDD.

[51]  Robert D. Nowak,et al.  Active Positive Semidefinite Matrix Completion: Algorithms, Theory and Applications , 2017, AISTATS.

[52]  V. Roshan Joseph,et al.  Support points , 2016, The Annals of Statistics.

[53]  Yao Xie,et al.  Maximum Entropy Low-Rank Matrix Recovery , 2017, IEEE Journal of Selected Topics in Signal Processing.

[54]  Yonina C. Eldar,et al.  Measurement Matrix Design for Phase Retrieval Based on Mutual Information , 2017, IEEE Transactions on Signal Processing.

[55]  C. F. Jeff Wu,et al.  cmenet: A New Method for Bi-Level Variable Selection of Conditional Main Effects , 2017, Journal of the American Statistical Association.