Majorization-Minimization algorithms for nonsmoothly penalized objective functions

The use of penalization, or regularization, has become common in high-dimensional statistical analysis, where an increasingly frequent goal is to simultaneously select important variables and estimate their effects. It has been shown by several authors that these goals can be achieved by minimizing some parameter-dependent "goodness-of-fit" function (e.g., a negative loglikelihood) subject to a penalization that promotes sparsity. Penalty functions that are singular at the origin have received substantial attention, arguably beginning with the Lasso penalty (62). The current literature tends to focus on specific combinations of dif- ferentiable goodness-of-fit functions and penalty functions singular at the origin. One result of this combined specificity has been a proliferation in the number of computational algorithms designed to solve fairly narrow classes of optimization problems involving objective functions that are not every- where continuously differentiable. In this paper, we propose a general class of algorithms for optimizing an extensive variety of nonsmoothly penalized objective functions that satisfy certain regularity conditions. The proposed framework utilizes the majorization-minimization (MM) algorithm as its core optimization engine. In the case of penalized regression models, the resulting algorithms employ iterated soft-thresholding, implemented com- ponentwise, allowing for fast and stable updating that avoids the need for inverting high-dimensional matrices. We establish convergence theory un- der weaker assumptions than previously considered in the statistical litera- ture. We also demonstrate the exceptional effectiveness of new acceleration methods, originally proposed for the EM algorithm, in this class of prob- lems. Simulation results and a microarray data example are provided to demonstrate the algorithm's capabilities and versatility. AMS 2000 subject classifications: Primary 65C60, 62J07; secondary 62J05, 62J12. Keywords and phrases: Convex optimization, iterative soft threshold- ing, Lasso penalty, minimax concave penalty, non-convex optimization, smoothly clipped absolute deviation penalty.