Restoring Definiteness via Shrinking, with an Application to Correlation Matrices with a Fixed Block

Indefinite approximations of positive semidefinite matrices arise in various data analysis applications involving covariance matrices and correlation matrices. We propose a method for restoring positive semidefiniteness of an indefinite matrix $M_0$ that constructs a convex linear combination $S(\alpha) = \alpha M_1 + (1-\alpha)M_0$ of $M_0$ and a positive semidefinite target matrix $M_1$. In statistics, this construction for improving an estimate $M_0$ by combining it with new information in $M_1$ is known as shrinking. We make no statistical assumptions about $M_0$ and define the optimal shrinking parameter as $\alpha_* = \min \{\alpha \in [0,1] : {$S(\alpha)$ is positive semidefinite}\}$. We describe three \alg s for computing $\alpha_*$. One algorithm is based on the bisection method, with the use of Cholesky factorization to test definiteness; a second employs Newton's method; and a third finds the smallest eigenvalue of a symmetric definite generalized eigenvalue problem. We show that weights that r...

[1]  Paul H. Kupiec,et al.  Stress Testing in a Value at Risk Framework , 1998 .

[2]  James Demmel,et al.  Applied Numerical Linear Algebra , 1997 .

[3]  N. Higham.  COMPUTING A NEAREST SYMMETRIC POSITIVE SEMIDEFINITE MATRIX , 1988 .

[4]  R. Rebonato,et al.  The most general methodology for creating a valid correlation matrix for risk management and option pricing purposes , 2000 .

[5]  Olivier Ledoit,et al.  A well-conditioned estimator for large-dimensional covariance matrices , 2004 .

[6]  Donald Hedeker,et al.  Simulation of massive public health data by power polynomials , 2012, Statistics in medicine.

[7]  Olivier Ledoit,et al.  Nonlinear Shrinkage Estimation of Large-Dimensional Covariance Matrices , 2011, 1207.5322.

[8]  Vineer Bhansali,et al.  Forecasting Portfolio Risk in Normal and Stressed Markets , 2001, nlin/0108022.

[9]  Eric Anderson,et al.  Simulating spatial cross-correlation in vehicular networks , 2014, 2014 IEEE Vehicular Networking Conference (VNC).

[10]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[11]  Defeng Sun,et al.  Correlation stress testing for value-at-risk: an unconstrained convex optimization approach , 2010, Comput. Optim. Appl..

[12]  Bruce C. Douglas,et al.  Experiments in Reconstructing Twentieth-Century Sea Levels , 2010 .

[13]  S. J. Devlin,et al.  Robust estimation and outlier detection with correlation coefficients , 1975 .

[14]  Y. Moon,et al.  Finding a positive definite linear combination of two Hermitian matrices , 1983 .

[15]  Nicholas J. Higham,et al.  A Preconditioned Newton Algorithm for the Nearest Correlation Matrix , 2010 .

[16]  B. McKay.  Honey , 1870, Hall's journal of health.

[17]  Abbas Seifi,et al.  A robust Kriging model for predicting accumulative outflow from a mature reservoir considering a new horizontal well , 2012 .

[18]  Craig Lucas.  Computing Nearest Covariance and Correlation Matrices , 2001 .

[19]  Nicholas J. Higham,et al.  INVERSE PROBLEMS NEWSLETTER , 1991 .

[20]  Nicholas J. Higham,et al.  An Improved Arc Algorithm for Detecting Definite Hermitian Pairs , 2009, SIAM J. Matrix Anal. Appl..

[21]  Olivier Ledoit,et al.  Honey, I Shrunk the Sample Covariance Matrix , 2003 .

[22]  Rüdiger Borsdorf.  A Newton Algorithm for the Nearest Correlation Matrix , 2007 .

[23]  Alec N. Kercheval.  OPTIMAL COVARIANCES IN RISK MODEL AGGREGATION , 2006 .

[24]  N. Higham.  Computing the nearest correlation matrix—a problem from finance , 2002 .

[25]  Cian O'Mahony,et al.  Statistical method to assess usual dietary intakes in the European population , 2014, Food additives & contaminants. Part A, Chemistry, analysis, control, exposure & risk assessment.

[26]  Javier Gómez-Aleixandre,et al.  Statistical characterization of aggregated wind power from small clusters of generators , 2014 .

[27]  N. Higham.  Analysis of the Cholesky Decomposition of a Semi-definite Matrix , 1990 .

[28]  L. Goldberg,et al.  On the Aggregation of Local Risk Models for Global Risk Management , 2005 .

[29]  K. Strimmer,et al.  Statistical Applications in Genetics and Molecular Biology A Shrinkage Approach to Large-Scale Covariance Matrix Estimation and Implications for Functional Genomics , 2011 .

[30]  Defeng Sun,et al.  A Quadratically Convergent Newton Method for Computing the Nearest Correlation Matrix , 2006, SIAM J. Matrix Anal. Appl..

[31]  R. Rebonato,et al.  The Most General Methodology to Create a Valid Correlation Matrix for Risk Management and Option Pricing Purposes , 2011 .

[32]  Thomas J. Latta.  Stress Testing in a Value at Risk Framework , 1999 .

[33]  Scott Gerlt,et al.  Exploiting the Relationship between Farm-Level Yields and County-Level Yields for Applied Analysis , 2014 .

[34]  Seung-Jean Kim,et al.  Condition‐number‐regularized covariance estimation , 2013, Journal of the Royal Statistical Society. Series B, Statistical methodology.

[35]  Kawee Numpacharoen,et al.  Weighted Average Correlation Matrices Method for Correlation Stress Testing and Sensitivity Analysis , 2013, The Journal of Derivatives.

[36]  S. Brotherstone,et al.  Estimating variance components and predicting breeding values for eventing disciplines and grades in sport horses. , 2012, Animal : an international journal of animal bioscience.

[37]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[38]  Olivier Ledoit,et al.  Improved estimation of the covariance matrix of stock returns with an application to portfolio selection , 2003 .

[39]  Nicholas J. Higham,et al.  Analysis of the Cholesky Method with Iterative Refinement for Solving the Symmetric Definite Generalized Eigenproblem , 2001, SIAM J. Matrix Anal. Appl..

[40]  H. Qi,et al.  An augmented Lagrangian dual approach for the H-weighted nearest correlation matrix problem , 2011 .

[41]  R. Kass,et al.  Shrinkage Estimators for Covariance Matrices , 2001, Biometrics.