Reduced rank regression via adaptive nuclear norm penalization.

We propose an adaptive nuclear norm penalization approach for low-rank matrix approximation, and use it to develop a new reduced rank estimation method for high-dimensional multivariate regression. The adaptive nuclear norm is defined as the weighted sum of the singular values of the matrix, and it is generally non-convex under the natural restriction that the weight decreases with the singular value. However, we show that the proposed non-convex penalized regression method has a global optimal solution obtained from an adaptively soft-thresholded singular value decomposition. The method is computationally efficient, and the resulting solution path is continuous. The rank consistency of and prediction/estimation performance bounds for the estimator are established for a high-dimensional asymptotic regime. Simulation studies and an application in genetics demonstrate its efficacy.

[1]  C. Eckart,et al.  The approximation of one matrix by another of lower rank , 1936 .

[2]  T. W. Anderson.  Estimating Linear Restrictions on Regression Coefficients for Multivariate Normal Distributions , 1951 .

[3]  M. Stone.  Cross-validation and multinomial prediction , 1974 .

[4]  A. Izenman.  Reduced-rank regression for the multivariate linear model , 1975 .

[5]  L. Mirsky.  A trace inequality of John von Neumann , 1975 .

[6]  J. Tukey,et al.  Variations of Box Plots , 1978 .

[7]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[8]  William W. Hager,et al.  Updating the Inverse of a Matrix , 1989, SIAM Rev..

[9]  David W. Lewis,et al.  Matrix theory , 1991 .

[10]  I. Johnstone,et al.  Adapting to Unknown Smoothness via Wavelet Shrinkage , 1995 .

[11]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[12]  G. Reinsel,et al.  Multivariate Reduced-Rank Regression: Theory and Applications , 1998 .

[13]  T. W. Anderson.  Asymptotic distribution of the reduced rank regression estimator under general conditions , 1999 .

[14]  Wenjiang J. Fu,et al.  Asymptotics for lasso-type estimators , 2000 .

[15]  ModelsThomas W. Yee.  Reduced-rank Vector Generalized Linear Models , 2000 .

[16]  M. Aldrin.  Multivariate Prediction using Softly Shrunk Reduced-Rank Regression , 2000 .

[17]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[18]  Christian A. Rees,et al.  Microarray analysis reveals a major direct role of DNA copy number alteration in the transcriptional program of human breast tumors , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[19]  T. W. Anderson.  Specification and misspecification in reduced rank regression , 2002 .

[20]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[21]  Ajay N. Jain,et al.  Genomic and transcriptional aberrations linked to breast cancer pathophysiologies. , 2006, Cancer cell.

[22]  H. Zou.  The Adaptive Lasso and Its Oracle Properties , 2006 .

[23]  M. Yuan,et al.  Dimension reduction and coefficient estimation in multivariate linear regression , 2007 .

[24]  Gene H. Golub,et al.  Generalized cross-validation as a method for choosing a good ridge parameter , 1979, Milestones in Matrix Computation.

[25]  Kung-Sik Chan,et al.  MULTIVARIATE REDUCED-RANK NONLINEAR TIME SERIES MODELING , 2007 .

[26]  H. Zou,et al.  One-step Sparse Estimates in Nonconcave Penalized Likelihood Models. , 2008, Annals of statistics.

[27]  J. Horowitz,et al.  Asymptotic properties of bridge estimators in sparse high-dimensional regression models , 2008, 0804.0693.

[28]  Jiaming Xu.  Reweighted Nuclear Norm Minimization for Matrix Completion , 2009 .

[29]  S. Yun,et al.  An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems , 2009 .

[30]  S. Yun,et al.  An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems , 2009 .

[31]  R. Tibshirani,et al.  A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis. , 2009, Biostatistics.

[32]  Hao Helen Zhang,et al.  ON THE ADAPTIVE ELASTIC-NET WITH A DIVERGING NUMBER OF PARAMETERS. , 2009, Annals of statistics.

[33]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[34]  Ji Zhu,et al.  Regularized Multivariate Regression for Identifying Master Predictors with Application to Integrative Genomics Study of Breast Cancer. , 2008, The annals of applied statistics.

[35]  V. Koltchinskii,et al.  Nuclear norm penalization and optimal rates for noisy low rank matrix completion , 2010, 1011.6256.

[36]  M. Wegkamp,et al.  Adaptive Rank Penalized Estimators in Multivariate Regression , 2010 .

[37]  A. Tsybakov,et al.  Estimation of high-dimensional low-rank matrices , 2009, 0912.5338.

[38]  W. C. Chan,et al.  Virtual CGH: an integrative approach to predict genetic abnormalities from gene expression microarray data applied in lymphoma , 2011, BMC Medical Genomics.

[39]  Martin J. Wainwright,et al.  Estimation of (near) low-rank matrices with noise and high-dimensional scaling , 2009, ICML.

[40]  Yiyuan She,et al.  Reduced Rank Vector Generalized Linear Models for Feature Extraction , 2010, 1007.3098.

[41]  O. Klopp.  Rank penalized estimators for high-dimensional matrices , 2011, 1104.1244.

[42]  M. Wegkamp,et al.  Optimal selection of reduced rank estimators of high-dimensional matrices , 2010, 1004.2995.

[43]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[44]  Ji Zhu,et al.  Reduced rank ridge regression and its kernel extensions , 2011, Stat. Anal. Data Min..

[45]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2008, Found. Comput. Math..

[46]  Renato D. C. Monteiro,et al.  Convex optimization methods for dimension reduction and coefficient estimation in multivariate linear regression , 2009, Mathematical Programming.

[47]  O. Stephens,et al.  Prediction of cytogenetic abnormalities with gene expression profiles. , 2012, Blood.

[48]  Kung-Sik Chan,et al.  Reduced rank stochastic regression with a sparse singular value decomposition , 2012 .

[49]  M. Wegkamp,et al.  Joint variable and rank selection for parsimonious estimation of high-dimensional matrices , 2011, 1110.3556.

[50]  Jianhua Z. Huang,et al.  Sparse Reduced-Rank Regression for Simultaneous Dimension Reduction and Variable Selection , 2012 .

[51]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .