Regularization of Case Specific Parameters: A New Approach for Improving Robustness and/or Efficiency of Statistical Methods

Regularization methods allow one to handle a variety of inferential problems where there are more covariates than cases. This allows one to consider a potentially enormous number of covariates for a problem. We exploit the power of these techniques, supersaturating models by augmenting the “natural” covariates in the problem with an additional indicator for each case in the data set. We attach a penalty term for these case-specific indicators which is designed to produce a desired effect. For regression methods with squared error loss, an 1 type penalty for case-specific parameters produces a regression which is robust to outliers and high leverage cases. Through this modification we have devised a robust LASSO which retains desirable property of the LASSO and performs better when outlying observations exist. For quantile regression methods, an 2 type penalty decreases the variance of the fit enough to overcome an increase in bias. The paradigm thus allows us to robustify procedures which lack robustness and to increase the efficiency of procedures which are robust. Including the case-specific parameters can be viewed as a modification of the current loss function to produce better estimator. For the LASSO with the squared error loss, the modification yields Huber’s loss. The check loss function in quantile regression is adjusted to be quadratic near its minimum. This modification produces an averaging effect near the target quantile thus more efficient quantile estimation in various settings. Applications ii to classification procedures such as logistic regression and support vector machines are also considered. Finally, a modification to cross validation through use of a new validation function in quantile regression is investigated. The new validation function makes use of the same adjusted check loss which is used for estimation.