How to select good classification methods

The past twenty years have seen a huge amount of highly innovative work on developing new classification tools. Classical methods such as discriminant analysis and logistic regression, and nonparametric approaches such as nearest neighbour and kernel methods, have been joined by tree methods, support vector machines, neural networks, and a large range of other methods. Many comparative studies have been conducted to assess the relative strengths and weaknesses of these various methods. However, when deciding which method to use, the single most important factor to consider is the criterion used to assess performance of the rule. A poor choice of performance criterion leads to a poor choice of method, and poor parameter estimates for that method. Worse, a poor choice of performance criterion gives the illusion that the chosen method is a good one, since it does well on the inappropriate criterion. The theoretical literature, the applications literature, and real practical commercial and scientific applications, pay too little attention to the choice of criterion, with the consequence that sub-optimal methods are often chosen. In this talk I look at performance criteria for classification rules, illustrating how poor choice of criterion can lead to very poor performance, and giving examples from a range of domains.