Fast and Flexible ADMM Algorithms for Trend Filtering

This article presents a fast and robust algorithm for trend filtering, a recently developed nonparametric regression tool. It has been shown that, for estimating functions whose derivatives are of bounded variation, trend filtering achieves the minimax optimal error rate, while other popular methods like smoothing splines and kernels do not. Standing in the way of a more widespread practical adoption, however, is a lack of scalable and numerically stable algorithms for fitting trend filtering estimates. This article presents a highly efficient, specialized alternating direction method of multipliers (ADMM) routine for trend filtering. Our algorithm is competitive with the specialized interior point methods that are currently in use, and yet is far more numerically robust. Furthermore, the proposed ADMM implementation is very simple, and, importantly, it is flexible enough to extend to many interesting related problems, such as sparse trend filtering and isotonic trend filtering. Software for our method is freely available, in both the C and R languages.

[1]  C. R. Deboor,et al.  A practical guide to splines , 1978 .

[2]  Carl de Boor,et al.  A Practical Guide to Splines , 1978, Applied Mathematical Sciences.

[3]  G. Wahba.  Spline models for observational data , 1990 .

[4]  R. Tibshirani,et al.  Generalized Additive Models , 1991 .

[5]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[6]  S. Geer,et al.  Locally adaptive regression splines , 1997 .

[7]  P. Davies,et al.  Local Extremes, Runs, Strings and Multiresolution , 2001 .

[8]  R. Tibshirani,et al.  Sparsity and smoothness via the fused lasso , 2005 .

[9]  Stephan Didas,et al.  Splines in Higher Order TV Regularization , 2006, International Journal of Computer Vision.

[10]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[11]  Quentin F. Stout,et al.  Unimodal regression via prefix isotonic regression , 2008, Comput. Stat. Data Anal..

[12]  Stephen P. Boyd,et al.  1 Trend Filtering , 2009, SIAM Rev..

[13]  Yiyuan She,et al.  Outlier Detection Using Nonconvex Penalized Regression , 2010, ArXiv.

[14]  R. Tibshirani,et al.  The solution path of the generalized lasso , 2010, 1005.1971.

[15]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[16]  Robert Tibshirani,et al.  Nearly-Isotonic Regression , 2011, Technometrics.

[17]  Nicholas A. Johnson,et al.  A Dynamic Programming Algorithm for the Fused Lasso and L 0-Segmentation , 2013 .

[18]  R. Tibshirani.  Adaptive piecewise polynomial estimation via trend filtering , 2013, 1304.2986.

[19]  Ryan J. Tibshirani,et al.  Efficient Implementations of the Generalized Lasso Dual Path Algorithm , 2014, ArXiv.

[20]  Alexander J. Smola,et al.  The Falling Factorial Basis and Its Statistical Applications , 2014, ICML.