Parallel Large Scale Feature Selection for Logistic Regression

In this paper we examine the problem of efficient feature evaluation for logistic regression on very large data sets. We present a new forward feature selection heuristic that ranks features by their estimated effect on the resulting model’s performance. An approximate optimization, based on backfitting, provides a fast and accurate estimate of each new feature’s coefficient in the logistic regression model. Further, the algorithm is highly scalable by parallelizing simultaneously over both features and records, allowing us to quickly evaluate billions of potential features even for very large data sets.

[1]  Rich Caruana,et al.  An empirical evaluation of supervised learning in high dimensions , 2008, ICML '08.

[2]  David Madigan,et al.  Algorithms for Sparse Linear Classifiers in the Massive Data Setting , 2008, J. Mach. Learn. Res..

[3]  Andrew McCallum,et al.  Efficiently Inducing Features of Conditional Random Fields , 2002, UAI.

[4]  John D. Lafferty,et al.  Inducing Features of Random Fields , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Andrew W. Moore,et al.  Making logistic regression a core data mining tool with TR-IRLS , 2005, Fifth IEEE International Conference on Data Mining (ICDM'05).

[6]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[7]  F. Fleuret.  Fast Binary Feature Selection with Conditional Mutual Information , 2004, J. Mach. Learn. Res..

[8]  TG David MadiganDepartment.  A novel feature selection score for text categorization , 2004 .

[9]  Ron Kohavi,et al.  Irrelevant Features and the Subset Selection Problem , 1994, ICML.

[10]  Lawrence Carin,et al.  Joint Classifier and Feature Optimization for Comprehensive Cancer Diagnosis Using Gene Expression Data , 2004, J. Comput. Biol..

[11]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[12]  J. Friedman,et al.  Projection Pursuit Regression , 1981 .

[13]  A. Wayne Whitney,et al.  A Direct Method of Nonparametric Measurement Selection , 1971, IEEE Transactions on Computers.

[14]  Yiming Yang,et al.  RCV1: A New Benchmark Collection for Text Categorization Research , 2004, J. Mach. Learn. Res..

[15]  Larry A. Rendell,et al.  The Feature Selection Problem: Traditional Methods and a New Algorithm , 1992, AAAI.

[16]  James Theiler,et al.  Online Feature Selection using Grafting , 2003, ICML.

[17]  Thomas G. Dietterich,et al.  Learning with Many Irrelevant Features , 1991, AAAI.

[18]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[19]  Huan Liu,et al.  Feature Selection for Classification , 1997, Intell. Data Anal..

[20]  R. Tibshirani,et al.  Linear Smoothers and Additive Models , 1989 .

[21]  Shigeo Abe,et al.  Modified backward feature selection by cross validation , 2005, ESANN.