Machine Learning: ECML 2000

Overfitting is often considered the central problem in machine learning and data mining. When good performance on training data is not enough to reliably predict good generalization, researchers and practitioners often invoke ”Occam’s razor” to select among hypotheses: prefer the simplest hypothesis consistent with the data. Occam’s razor has a long history in science, but a mass of recent evidence suggests that in most cases it is outperformed by methods that deliberately produce more complex models. The poor performance of Occam’s razor can be largely traced to its failure to account for the search process by which hypotheses are obtained: by effectively assuming that the hypothesis space is exhaustively searched, complexity-based methods tend to over-penalize large spaces. This talk describes how information about the search process can be taken into account when evaluating hypotheses. The expected generalization error of a hypothesis is computed as a function of the search steps leading to it. Two variations of this ”processoriented” approach have yielded significant improvements in the accuracy of a rule learner. Process-oriented evaluation leads to the seemingly paradoxical conclusion that the same hypothesis will have different expected generalization errors depending on how it was generated. I believe that this is as it should be, and that a corresponding shift in our way of thinking about inductive learning is required.