Variable selection methods in regression: Ignorable problem, outing notable solution

Variable selection in regression – identifying the best subset among many variables to include in a model – is arguably the hardest part of model building. Many variable selection methods exist. Many statisticians know them, but few know they produce poorly performing models. Some variable selection methods are a miscarriage of statistics because they are developed by, in effect, debasing sound statistical theory. The purpose of this article is two-fold: (1) to re-examine the scope of the literature addressing the weaknesses of variable selection methods and (2) to re-enliven a possible solution to defining a better-performing regression model. To achieve this goal in practice, the article has two objectives: (1) to review five widely used variable selection methods, itemize some of their weaknesses and consider why they are used; and (2) to present Tukey's Exploratory Data Analysis (EDA) in the context of a natural seven-step cycle. Newcomers to Tukey's EDA should consider the seven-step cycle introduced in the narrative of Tukey's analytic philosophy. John W. Tukey (16 June 1915 – 26 July 2000) was a significant contributor to the field of statistics, but was a humble, unpretentious man, as he always considered himself a data analyst. Tukey's seminal book, Exploratory Data Analysis is known by the book's unique initialed title, EDA.