Improving Classification under Changes in Class and Within-Class Distributions

The fundamental assumption that training and operational data come from the same probability distribution, which is the basis of most learning algorithms, is often not satisfied in practice. Several algorithms have been proposed to cope with classification problems where the class priors may change after training, but they can show a poor performance when the class conditional data densities also change. In this paper, we propose a re-estimation algorithm that makes use of unlabeled operational data to adapt the classifier behavior to changing scenarios. We assume that (a) the classes may be decomposed in several (unknown) subclasses, and (b) the prior subclass probabilities may change after training. Experimental results with practical applications show an improvement over an adaptive method based on class priors, while preserving a similar performance when there are no subclass changes.

[1]  Peter A. Flach,et al.  A Response to Webb and Ting’s On the Application of ROC Analysis to Predict Classification Performance Under Varying Class Distributions , 2005, Machine Learning.

[2]  Marco Saerens,et al.  Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure , 2002, Neural Computation.

[3]  Domingo Docampo,et al.  Growing Gaussian mixtures network for classification applications , 1999, Signal Process..

[4]  Geoffrey I. Webb,et al.  On the Application of ROC Analysis to Predict Classification Performance Under Varying Class Distributions , 2005, Machine Learning.

[5]  R. Holte Elaboration on Two Points Raised in “Classifier Technology and the Illusion of Progress” , 2006, math/0606455.

[6]  Tom Fawcett,et al.  Robust Classification for Imprecise Environments , 2000, Machine Learning.

[7]  Tom Fawcett,et al.  Robust Classification Systems for Imprecise Environments , 1998, AAAI/IAAI.

[8]  Jesús Cid-Sueiro,et al.  A model selection algorithm for a posteriori probability estimation with neural networks , 2005, IEEE Transactions on Neural Networks.

[9]  Jesús Cid-Sueiro,et al.  Minimax Regret Classifier for Imprecise Class Distributions , 2007, J. Mach. Learn. Res..

[10]  Motoaki Kawanabe,et al.  Asymptotic Bayesian generalization error when training and test distributions are different , 2007, ICML '07.

[11]  H. Shimodaira,et al.  Improving predictive inference under covariate shift by weighting the log-likelihood function , 2000 .