Comparison of approaches for estimating reliability of individual regression predictions

The paper compares different approaches to estimate the reliability of individual predictions in regression. We compare the sensitivity-based reliability estimates developed in our previous work with four approaches found in the literature: variance of bagged models, local cross-validation, density estimation, and local modeling. By combining pairs of individual estimates, we compose a combined estimate that performs better than the individual estimates. We tested the estimates by running data from 28 domains through eight regression models: regression trees, linear regression, neural networks, bagging, support vector machines, locally weighted regression, random forests, and generalized additive model. The results demonstrate the potential of a sensitivity-based estimate, as well as the local modeling of prediction error with regression trees. Among the tested approaches, the best average performance was achieved by estimation using the bagging variance approach, which achieved the best performance with neural networks, bagging and locally weighted regression.

[1]  Avrim Blum,et al.  The Bottleneck , 2021, Monopsony Capitalism.

[2]  Tom Heskes,et al.  Practical Confidence and Prediction Intervals , 1996, NIPS.

[3]  Andreas S. Weigend,et al.  Predictions with Confidence Intervals ( Local Error Bars ) , 1994 .

[4]  Virginia R. de Sa,et al.  Learning Classification with Unlabeled Data , 1993, NIPS.

[5]  Alexander Gammerman,et al.  Learning by Transduction , 1998, UAI.

[6]  Vladimir Vapnik,et al.  The Nature of Statistical Learning , 1995 .

[7]  Steven D. Whitehead,et al.  A Complexity Analysis of Cooperative Mechanisms in Reinforcement Learning , 1991, AAAI.

[8]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 2019, Texts in Computer Science.

[9]  Dana Ron,et al.  Algorithmic Stability and Sanity-Check Bounds for Leave-One-Out Cross-Validation , 1997, Neural Computation.

[10]  W S McCulloch,et al.  A logical calculus of the ideas immanent in nervous activity , 1990, The Philosophy of Artificial Intelligence.

[11]  Tom Michael Mitchell,et al.  The Role of Unlabeled Data in Supervised Learning , 2004 .

[12]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[13]  Jan Mendling,et al.  Detection and prediction of errors in EPCs of the SAP reference model , 2008, Data Knowl. Eng..

[14]  Igor Kononenko,et al.  Reliable Classifications with Machine Learning , 2002, ECML.

[15]  Stefano Tarantola,et al.  Introduction to Sensitivity Analysis , 2008 .

[16]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[17]  M. Pontil.  Leave-one-out error and stability of learning algorithms with applications , 2002 .

[18]  Stefan Schaal,et al.  Natural Actor-Critic , 2003, Neurocomputing.

[19]  Johan A. K. Suykens,et al.  Advances in learning theory : methods, models and applications , 2003 .

[20]  Igor Kononenko,et al.  Machine Learning and Data Mining: Introduction to Principles and Algorithms , 2007 .

[21]  Stefan Schaal,et al.  Assessing the Quality of Learned Local Models , 1993, NIPS.

[22]  Kevin W. Bowyer,et al.  Combination of multiple classifiers using local accuracy estimates , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[23]  Alexander Gammerman,et al.  Transduction with Confidence and Credibility , 1999, IJCAI.

[24]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[25]  Zhi-Hua Zhou,et al.  Semisupervised Regression with Cotraining-Style Algorithms , 2007, IEEE Transactions on Knowledge and Data Engineering.

[26]  David A. Cohn,et al.  Active Learning with Statistical Models , 1996, NIPS.

[27]  Z. Bosnic,et al.  Evaluation of prediction reliability in regression using the transduction principle , 2003, The IEEE Region 8 EUROCON 2003. Computer as a Tool..

[28]  Igor Kononenko,et al.  Estimation of individual prediction reliability using the local sensitivity analysis , 2008, Applied Intelligence.

[29]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[30]  Yan Zhou,et al.  Enhancing Supervised Learning with Unlabeled Data , 2000, ICML.

[31]  Tsai-Hung Fan,et al.  Tests and variables selection on regression analysis for massive datasets , 2007, Data Knowl. Eng..

[32]  Susan A. Murphy,et al.  Monographs on statistics and applied probability , 1990 .

[33]  Christopher G. Atkeson,et al.  Constructive Incremental Learning from Only Local Information , 1998, Neural Computation.

[34]  Jack P. C. Kleijnen.  Experimental Design for Sensitivity Analysis of Simulation Models , 2001 .

[35]  Dale Schuurmans,et al.  Data perturbation for escaping local maxima in learning , 2002, AAAI/IAAI.

[36]  André Elisseeff,et al.  Algorithmic Stability and Generalization Performance , 2000, NIPS.

[37]  Mauro Birattari,et al.  Local learning for data analysis , 1998 .

[38]  Bernard W. Silverman,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[39]  R. Tibshirani,et al.  Model Search by Bootstrap “Bumping” , 1999 .

[40]  Padraig Cunningham,et al.  Confidence and prediction intervals for neural network ensembles , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[41]  Yuh-Jye Lee,et al.  epsilon-SSVR: A Smooth Support Vector Machine for epsilon-Insensitive Regression , 2005, IEEE Trans. Knowl. Data Eng..

[42]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[43]  Igor Kononenko,et al.  Estimation of Regressor Reliability , 2008 .

[44]  Wai Lam,et al.  Utilizing hierarchical feature domain values for prediction , 2007, Data Knowl. Eng..

[45]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[46]  Alan Y. Chiang,et al.  Generalized Additive Models: An Introduction With R , 2007, Technometrics.

[47]  Matthias Seeger,et al.  Learning from Labeled and Unlabeled Data , 2010, Encyclopedia of Machine Learning.

[48]  Gunnar Rätsch,et al.  Learning to Predict the Leave-One-Out Error of Kernel Based Classifiers , 2001, ICANN.

[49]  Simon N. Wood,et al.  Generalized Additive Models , 2006, Annual Review of Statistics and Its Application.

[50]  Zhi-Hua Zhou,et al.  Semi-Supervised Regression with Co-Training Style Algorithms , 2007 .

[51]  David A. Landgrebe,et al.  Fast Parzen Density Estimation Using Clustering-Based Branch and Bound , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[52]  S. Hochreiter,et al.  REINFORCEMENT DRIVEN INFORMATION ACQUISITION IN NONDETERMINISTIC ENVIRONMENTS , 1995 .

[53]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[54]  Martin Crowder,et al.  Statistical Analysis of Reliability Data , 1991 .

[55]  David A. Cohn,et al.  Training Connectionist Networks with Queries and Selective Sampling , 1989, NIPS.

[56]  David H. Wolpert,et al.  Stacked generalization , 1992, Neural Networks.

[57]  André Elisseeff,et al.  Stability and Generalization , 2002, J. Mach. Learn. Res..

[58]  B. Silverman.  Density estimation for statistics and data analysis , 1986 .

[59]  Fabio Roli,et al.  Dynamic classifier selection based on multiple classifier behaviour , 2001, Pattern Recognit..

[60]  Vladimir Vovk,et al.  Ridge Regression Confidence Machine , 2001, International Conference on Machine Learning.

[61]  Robert Tibshirani,et al.  Model Search and Inference By Bootstrap "bumping , 1995 .

[62]  Matthew P. Wand,et al.  Kernel Smoothing , 1995 .

[63]  William I. Gasarch,et al.  Book Review: An introduction to Kolmogorov Complexity and its Applications Second Edition, 1997 by Ming Li and Paul Vitanyi (Springer (Graduate Text Series)) , 1997, SIGACT News.

[64]  Stefano Tarantola,et al.  Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models , 2004 .