On the degrees of freedom in shrinkage estimation

We study the degrees of freedom in shrinkage estimation of regression coefficients. Generalizing the idea of the Lasso, we consider the problem of estimating the coefficients by minimizing the sum of squares with the constraint that the coefficients belong to a closed convex set. Based on a differential geometric approach, we derive an unbiased estimator of the degrees of freedom for this estimation method, under a smoothness assumption on the boundary of the closed convex set. The result presented in this paper is applicable to estimation with a wide class of constraints. As an application, we obtain a C"p type criterion and AIC for selecting tuning parameters.

[1]  R. Tibshirani,et al.  Sparsity and smoothness via the fused lasso , 2005 .

[2]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[3]  A. Gorin.  ON THE VOLUME OF TUBES , 1983 .

[4]  M. Yuan,et al.  Model selection and estimation in regression with grouped variables , 2006 .

[5]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[6]  S. Pandey,et al.  What Are Degrees of Freedom , 2008 .

[7]  Beryl Rawson,et al.  Degrees of Freedom , 2010 .

[8]  M. R. Osborne,et al.  On the LASSO and its Dual , 2000 .

[9]  R. Tibshirani,et al.  On the “degrees of freedom” of the lasso , 2007, 0712.0881.

[10]  Wenjiang J. Fu.  Penalized Regressions: The Bridge versus the Lasso , 1998 .

[11]  Akimichi Takemura,et al.  Shrinkage Estimation towards a Closed Convex Set with a Smooth Boundary , 2000 .

[12]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[13]  C. L. Mallows.  Some comments on C_p , 1973 .

[14]  Ji Zhu,et al.  Quantile Regression in Reproducing Kernel Hilbert Spaces , 2007 .

[15]  Akimichi Takemura,et al.  Shrinkage to Smooth Non-Convex Cone: Principal Component Analysis as Stein Estimation , 1998 .

[16]  R. Schneider.  Convex Bodies: The Brunn–Minkowski Theory: Minkowski addition , 1993 .

[17]  C. Mallows.  More comments on C p , 1995 .

[18]  Mary C. Meyer,et al.  ON THE DEGREES OF FREEDOM IN SHAPE-RESTRICTED REGRESSION , 2000 .

[19]  C. Stein.  Estimation of the Mean of a Multivariate Normal Distribution , 1981 .

[20]  Jesse Freeman,et al.  in Morse theory, , 1999 .