How Many Iterations are Sufficient for Semiparametric Estimation

A common practice in obtaining a semiparametric efficient estimate is through iteratively maximizing the (penalized) log-likelihood w.r.t. its Euclidean parameter and functional nuisance parameter via Newton-Raphson algorithm. The purpose of this paper is to provide a formula in calculating the minimal number of iterations $k^\ast$ needed to produce an efficient estimate $\hat\theta_n^{(k^\ast)}$ from a theoretical point of view. We discover that (a) $k^\ast$ depends on the convergence rates of the initial estimate and nuisance estimate; (b) more than $k^\ast$ iterations, i.e., $k$, will only improve the higher order asymptotic efficiency of $\hat\theta_n^{(k)}$; (c) $k^\ast$ iterations are also sufficient for recovering the estimation sparsity in high dimensional data. These general conclusions hold, in particular, when the nuisance parameter is not estimable at root-n rate, and apply to semiparametric models estimated under various regularizations, e.g., kernel or penalized estimation. This paper provides a first general theoretical justification for the "one-/two-step iteration" phenomena observed in the literature, and may be useful in reducing the bootstrap computational cost for the semiparametric models.

[1]  Anton Schick,et al.  Root-n-consistent and efficient estimation in semiparametric additive regression models , 1996 .

[2]  T. Severini,et al.  Quasi-Likelihood Estimation in Semiparametric Models , 1994 .

[3]  Peter Craven,et al.  Smoothing noisy data with spline functions , 1978 .

[4]  Jon A. Wellner,et al.  TWO LIKELIHOOD-BASED SEMIPARAMETRIC ESTIMATION METHODS FOR PANEL COUNT DATA WITH COVARIATES , 2005, math/0509132.

[5]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[6]  Adonis Yatchew,et al.  An elementary estimator of the partial linear model , 1997 .

[7]  Jianqing Fan,et al.  Generalized Partially Linear Single-Index Models , 1997 .

[8]  Jianhua Z. Huang,et al.  Efficient estimation in marginal partially linear models for longitudinal/clustered data using splines , 2007 .

[9]  Susan A. Murphy,et al.  Observed information in semi-parametric models , 1999 .

[10]  Nancy E. Heckman,et al.  Spline Smoothing in a Partly Linear Model , 1986 .

[11]  K. Roeder,et al.  A Semiparametric Mixture Approach to Case-Control Studies with Errors in Covariables , 1996 .

[12]  M. Kosorok.  Introduction to Empirical Processes and Semiparametric Inference , 2008 .

[13]  Donald W. K. Andrews,et al.  Nonparametric Kernel Estimation for Semiparametric Models , 1995, Econometric Theory.

[14]  Jianqing Fan,et al.  Local polynomial kernel regression for generalized linear models and quasi-likelihood functions , 1995 .

[15]  Jianhua Z. Huang,et al.  Bootstrap consistency for general semiparametric $M$-estimation , 2009, 0906.1310.

[16]  A. W. van der Vaart,et al.  On Profile Likelihood , 2000 .

[17]  P. Janssen,et al.  Rate of Convergence of One- and Two-Step $M$-Estimators with Applications to Maximum Likelihood and Pitman Estimators , 1985 .

[18]  J. Staniswalis.  The Kernel Estimate of a Regression Function in Likelihood-Based Models , 1989 .

[19]  A. Schick.  On Asymptotically Efficient Estimation in Semiparametric Models , 1986 .

[20]  P. Robinson,et al.  The stochastic difference between econometric statistics , 1988 .

[21]  Donald W. K. Andrews,et al.  Higher‐Order Improvements of a Computationally Attractive k‐Step Bootstrap for Extremum Estimators , 2002 .

[22]  Xiaohong Chen.  Chapter 76 Large Sample Sieve Estimation of Semi-Nonparametric Models , 2007 .

[23]  M. Kosorok,et al.  HIGHER ORDER SEMIPARAMETRIC FREQUENTIST INFERENCE WITH THE PROFILE SAMPLER , 2006, math/0605134.

[24]  M. Banerjee,et al.  Semiparametric binary regression models under shape constraints with an application to Indian schooling data , 2009 .

[25]  Florentina Bunea,et al.  Covariate selection for semiparametric hazard function regression models , 2005 .

[26]  Grace Wahba,et al.  Spline Models for Observational Data , 1990 .

[27]  P. Speckman.  Kernel smoothing in partial linear models , 1988 .

[28]  M. Kosorok,et al.  The penalized profile sampler , 2007, J. Multivar. Anal..

[29]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[30]  Jian Huang,et al.  Efficient Estimation for the Cox Model with Interval Censoring Efficient Estimation for the Cox Model with Interval Censoring , 1994 .

[31]  W. Wong,et al.  Profile Likelihood and Conditionally Parametric Models , 1992 .

[32]  K. Do,et al.  Efficient and Adaptive Estimation for Semiparametric Models. , 1994 .

[33]  Xiaohong Chen,et al.  Efficient Estimation of Models with Conditional Moment Restrictions Containing Unknown Functions , 2003 .

[34]  Susan A. Murphy,et al.  MLE in the proportional odds model , 1996 .

[35]  Chris A. J. Klaassen,et al.  Consistent Estimation of the Influence Function of Locally Asymptotically Linear Estimators , 1987 .

[36]  Sara van de Geer,et al.  Penalized quasi-likelihood estimation in partial linear models , 1997 .

[37]  M. Kosorok,et al.  General frequentist properties of the posterior profile distribution , 2006, math/0612191.

[38]  I. Keilegom,et al.  Estimation of a semiparametric transformation model , 2008, 0804.0719.

[39]  Anton Schick,et al.  A note on the construction of asymptotically linear estimators , 1987 .

[40]  Guang Cheng,et al.  Semiparametric additive isotonic regression , 2009 .

[41]  P. Bickel.  On Adaptive Estimation , 1982 .