Adaptive Mixtures of Regressions: Improving Predictive Inference when Population has Changed

The present work investigates the estimation of regression mixtures when population has changed between the training and the prediction stages. Two approaches are proposed: a parametric approach modeling the relationship between dependent variables of both populations, and a Bayesian approach in which the priors on the prediction population depend on the mixture regression parameters of the training population. The relevance of both approaches is illustrated on simulations and on an environmental dataset.

[1]  Hongtu Zhu,et al.  Hypothesis testing in mixture regression models , 2004 .

[2]  Christian P. Robert,et al.  The Bayesian choice , 1994 .

[3]  Masashi Sugiyama,et al.  Active Learning in Approximately Linear Regression Based on Conditional Expectation of Generalization Error , 2006, J. Mach. Learn. Res..

[4]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[5]  Christian Hennig,et al.  Identifiablity of Models for Clusterwise Linear Regression , 2000, J. Classif..

[6]  Qiang Yang,et al.  Topic-bridged PLSA for cross-domain text classification , 2008, SIGIR '08.

[7]  John Blitzer,et al.  Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification , 2007, ACL.

[8]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[9]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[10]  C. Robert,et al.  Estimating Mixtures of Regressions , 2003 .

[11]  C. Bouveyron,et al.  Adaptive linear models for regression: Improving prediction when population has changed , 2010, Pattern Recognit. Lett..

[12]  Neil D. Lawrence,et al.  Learning to learn with the informative vector machine , 2004, ICML.

[13]  C. Biernacki,et al.  A Generalized Discriminant Rule When Training Population and Test Population Differ on Their Descriptive Parameters , 2002, Biometrics.

[14]  Steven D. Brown,et al.  Transfer of multivariate calibration models: a review , 2002 .

[15]  M. Stephens.  Dealing with label switching in mixture models , 2000 .

[16]  David M. Allen,et al.  The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction , 1974 .

[17]  P. Green,et al.  On Bayesian Analysis of Mixtures with an Unknown Number of Components (with discussion) , 1997 .

[18]  Ralf Der,et al.  Classification in the Information Age , 1999 .

[19]  B. Kowalski,et al.  Multivariate instrument standardization , 1991 .

[20]  F. Leisch.  FlexMix: A general framework for finite mixture models and latent class regression in R , 2004 .

[21]  Massimiliano Pontil,et al.  Regularized multi--task learning , 2004, KDD.

[22]  Christophe Biernacki,et al.  Initializing EM using the properties of its trajectories in Gaussian mixtures , 2004, Stat. Comput..

[23]  M. Newton,et al.  Estimating the Integrated Likelihood via Posterior Simulation Using the Harmonic Mean Identity , 2006 .

[24]  Masashi Sugiyama,et al.  Input-dependent estimation of generalization error under covariate shift , 2005 .

[25]  H. Shimodaira,et al.  Improving predictive inference under covariate shift by weighting the log-likelihood function , 2000 .

[26]  J. T. Magee,et al.  On mass spectrometer instrument standardization and interlaboratory calibration transfer using neural networks , 1997 .

[27]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[28]  Klaus-Robert Müller,et al.  Covariate Shift Adaptation by Importance Weighted Cross Validation , 2007, J. Mach. Learn. Res..

[29]  C. Robert,et al.  Computational and Inferential Difficulties with Mixture Posterior Distributions , 2000 .

[30]  Kenneth W. Tobin,et al.  Content-Based Image Retrieval for Semiconductor Process Characterization , 2002, EURASIP J. Adv. Signal Process..

[31]  Kristine A. Bertness,et al.  Noise reduction in optical in situ measurements for molecular beam epitaxy by substrate wobble normalization , 1998 .

[32]  P. Green,et al.  Corrigendum: On Bayesian analysis of mixtures with an unknown number of components , 1997 .

[33]  Ivor W. Tsang,et al.  Domain Adaptation via Transfer Component Analysis , 2009, IEEE Transactions on Neural Networks.

[34]  Heping Zhang,et al.  Hypothesis Testing in Mixture Regression Models ( Mathematical Details ) , 2004 .

[35]  C. Robert,et al.  Estimation of Finite Mixture Distributions Through Bayesian Sampling , 1994 .

[36]  J. Q. Smith,et al.  1. Bayesian Statistics 4 , 1993 .

[37]  Masashi Sugiyama,et al.  Mixture Regression for Covariate Shift , 2006, NIPS.

[38]  Jiahua Chen,et al.  Variable Selection in Finite Mixture of Regression Models , 2007 .

[39]  Julien Jacques,et al.  Extension of model-based classification for binary data when training and test populations differ , 2010 .

[40]  S. Goldfeld,et al.  A Markov model for switching regressions , 1973 .