Bayes Optimal Hyperplanes ! Maximal Margin Hyperplanes

Maximal margin classifiers are a core technology in modern machine learning. They have strong theoretical justifications and have shown empirical successes. We provide an alternative justification for maximal margin hyperplane classifiers by relating them to Bayes optimal classifiers that use Parzen windows estimations with Gaussian kernels. For any value of the smoothing parameter (the width of the Gaussian kernels), the Bayes optimal classifier defines a density over the space of instances. We define the Bayes optimal hyperplaneto be the hyperplane decision boundary that gives lowest probability of classification error relative to this densit y. We show that, for linearly separable data, as we reduce the smoothing parameter to zero, a hyperplane is the Bayes optimal hyperplane if and only if it is the maximal margin hyperplane. We also analyze the behavior of the Bayes optimal hyperplane for non-linearly-separable data, showing that it has a very natural form. We explore the idea of using the hyperplane that is optimal relative to a density with some small non-zero kernel width, and present some promising preliminary results.