On risk bounds in isotonic and other shape restricted regression problems

We consider the problem of estimating an unknown $\theta\in {\mathbb{R}}^n$ from noisy observations under the constraint that $\theta$ belongs to certain convex polyhedral cones in ${\mathbb{R}}^n$. Under this setting, we prove bounds for the risk of the least squares estimator (LSE). The obtained risk bound behaves differently depending on the true sequence $\theta$ which highlights the adaptive behavior of $\theta$. As special cases of our general result, we derive risk bounds for the LSE in univariate isotonic and convex regression. We study the risk bound in isotonic regression in greater detail: we show that the isotonic LSE converges at a whole range of rates from $\log n/n$ (when $\theta$ is constant) to $n^{-2/3}$ (when $\theta$ is uniformly increasing in a certain sense). We argue that the bound presents a benchmark for the risk of any estimator in isotonic regression by proving nonasymptotic local minimax lower bounds. We prove an analogue of our bound for model misspecification where the true $\theta$ is not necessarily nondecreasing.

[1]  H. D. Brunk,et al.  AN EMPIRICAL DISTRIBUTION FUNCTION FOR SAMPLING WITH INCOMPLETE INFORMATION , 1955 .

[2]  H. D. Brunk.  Maximum Likelihood Estimates of Monotone Parameters , 1955 .

[3]  U. Grenander.  On the theory of mortality measurement , 1956 .

[4]  F. T. Wright.  The Asymptotic Behavior of Monotone Regression Estimates , 1981 .

[5]  Piet Groeneboom,et al.  University of Washington , 2019, The Grants Register 2020.

[6]  P. Groeneboom.  Estimating a monotone density , 1984 .

[7]  F. T. Wright,et al.  Order restricted statistical inference , 1988 .

[8]  S. Geer.  Estimating a Regression Function , 1990 .

[9]  L. Birge,et al.  The Grenader Estimator: A Nonasymptotic Approach , 1989 .

[10]  R. Durrett.  Probability: Theory and Examples , 1993 .

[11]  P. Massart,et al.  Rates of convergence for minimum contrast estimators , 1993 .

[12]  S. Geer.  Hellinger-Consistency of Certain Nonparametric Maximum Likelihood Estimators , 1993 .

[13]  Yazhen Wang,et al.  The L2risk of an isotonic estimate , 1996 .

[14]  P. Massart,et al.  Risk bounds for model selection via penalization , 1999 .

[15]  Alexander Schrijver,et al.  Theory of linear and integer programming , 1986, Wiley-Interscience series in discrete mathematics and optimization.

[16]  Asymptotic behavior of the grenander estimator at density flat regions , 1999 .

[17]  Mary C. Meyer,et al.  ON THE DEGREES OF FREEDOM IN SHAPE-RESTRICTED REGRESSION , 2000 .

[18]  Cun-Hui Zhang.  Risk bounds in isotonic regression , 2002 .

[19]  Dimitri P. Bertsekas,et al.  Convex Analysis and Optimization , 2003 .

[20]  M. Rudelson,et al.  Sparse reconstruction by convex relaxation: Fourier and Gaussian measurements , 2006, 2006 40th Annual Conference on Information Sciences and Systems.

[21]  Eric Cator,et al.  Adaptivity and optimality of the monotone least-squares estimator , 2008 .

[22]  Alexandre B. Tsybakov,et al.  Introduction to Nonparametric Estimation , 2008, Springer series in statistics.

[23]  M. Stojnic.  Various thresholds for $\ell_1$-optimization in compressed sensing , 2009 .

[24]  Pablo A. Parrilo,et al.  The Convex Geometry of Linear Inverse Problems , 2010, Foundations of Computational Mathematics.

[25]  A. Tsybakov,et al.  Sparse Estimation by Exponential Weighting , 2011, 1108.5116.

[26]  H. Jankowski.  Convergence of linear functionals of the Grenander estimator under misspecification , 2012, 1207.6614.

[27]  Adityanand Guntuboyina,et al.  Global risk bounds and adaptation in univariate convex regression , 2013, 1305.1648.

[28]  Joel A. Tropp,et al.  Living on the edge: A geometric theory of phase transitions in convex optimization , 2013, ArXiv.

[29]  Joel A. Tropp,et al.  Living on the edge: phase transitions in convex programs with random data , 2013, 1303.6672.

[30]  T. Cai,et al.  A Framework For Estimation of Convex Functions , 2015 .

[31]  Babak Hassibi,et al.  Asymptotically Exact Denoising in Relation to Compressed Sensing , 2013, ArXiv.