Reliable prediction intervals with regression neural networks

This paper proposes an extension to conventional regression neural networks (NNs) for replacing the point predictions they produce with prediction intervals that satisfy a required level of confidence. Our approach follows a novel machine learning framework, called Conformal Prediction (CP), for assigning reliable confidence measures to predictions without assuming anything more than that the data are independent and identically distributed (i.i.d.). We evaluate the proposed method on four benchmark datasets and on the problem of predicting Total Electron Content (TEC), which is an important parameter in trans-ionospheric links; for the latter we use a dataset of more than 60000 TEC measurements collected over a period of 11 years. Our experimental results show that the prediction intervals produced by our method are both well calibrated and tight enough to be useful in practice.

[1]  Alexander Gammerman,et al.  Transduction with Confidence and Credibility , 1999, IJCAI.

[2]  Iwona Stanislawska,et al.  Nowcasting, forecasting and warning for ionospheric propagation: tools and methods , 2004 .

[3]  Beichen Zhang,et al.  Preliminary studies on ionospheric forecasting in China and its surrounding area , 2005 .

[4]  Lazaros S. Iliadis,et al.  An Artificial Neural Network model for mountainous water-resources management: The case of Cyprus mountainous watersheds , 2007, Environ. Model. Softw..

[5]  E. O. Oyeyemi,et al.  Towards the development of a new global foF2 empirical model using neural networks , 2004 .

[6]  George C. Anastassopoulos,et al.  A non-symbolic implementation of abdominal pain estimation in childhood , 2008, Inf. Sci..

[7]  Takashi Maruyama,et al.  Regional reference total electron content model over Japan based on neural network mapping techniques , 2007 .

[8]  Henrik Lundstedt,et al.  A neural network-based local model for prediction of geomagnetic disturbances , 2001 .

[9]  H. Gleisner,et al.  Predicting geomagnetic storms from solar-wind data using time-delay neural networks , 1996 .

[10]  Harris Papadopoulos,et al.  Reliable Confidence Intervals for Software Effort Estimation , 2009, AIAI Workshops.

[11]  Valeriy V. Gavrishchaka,et al.  Support vector machine as an efficient tool for high‐dimensional data processing: Application to substorm forecasting , 2001 .

[12]  Haris Haralambous,et al.  A local Total Electron Content neural network model over Cyprus , 2010, 2010 4th International Symposium on Communications, Control and Signal Processing (ISCCSP).

[13]  Alexander Gammerman,et al.  Learning by Transduction , 1998, UAI.

[14]  Harris Papadopoulos,et al.  Inductive Confidence Machines for Regression , 2002, ECML.

[15]  Harris Papadopoulos,et al.  Assessment of Stroke Risk Based on Morphological Ultrasound Image Analysis with Conformal Prediction , 2010, AIAI.

[16]  Harris Papadopoulos,et al.  Normalized nonconformity measures for regression Conformal Prediction , 2008 .

[17]  Srdjan S. Stankovic,et al.  Ionospheric forecasting technique by artificial neural network , 1998 .

[18]  G. Shafer,et al.  Algorithmic Learning in a Random World , 2005 .

[19]  Harris Papadopoulos,et al.  Regression Conformal Prediction with Nearest Neighbours , 2014, J. Artif. Intell. Res..

[20]  Jin Zhang,et al.  Recognition of hypoxia EEG with a preset confidence level based on EEG analysis , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[21]  Harris Papadopoulos,et al.  Reliable diagnosis of acute abdominal pain with conformal prediction , 2009 .

[22]  Ljiljana R. Cander,et al.  Total electron content - A key parameterin propagation: measurement and usein ionospheric imaging , 2004 .

[23]  Vladimir Vovk,et al.  Comparing the Bayes and Typicalness Frameworks , 2001, ECML.

[24]  Lazaros S. Iliadis,et al.  Application of fuzzy T-norms towards a new Artificial Neural Networks' evaluation framework: A case from wood industry , 2008, Inf. Sci..

[25]  D. S. Broomhead,et al.  Prediction of the hourly ionospheric parameter ƒ0 F 2 using a novel nonlinear interpolation technique to cope with missing data points , 2001 .

[26]  Henrik Lundstedt,et al.  Neural networks and predictions of solar-terrestrial effects , 1992 .

[27]  Harris Papadopoulos,et al.  Inductive Conformal Prediction: Theory and Application to Neural Networks , 2008 .

[28]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[29]  K. Macpherson.  Neural network computation techniques applied to solar activity prediction , 1993 .

[30]  Harris Papadopoulos,et al.  Reliable Confidence Measures for Medical Diagnosis With Evolutionary Algorithms , 2011, IEEE Transactions on Information Technology in Biomedicine.

[31]  Alexander Gammerman,et al.  Transductive Confidence Machines for Pattern Recognition , 2002, ECML.

[32]  Milan Milosavljević,et al.  Ionospheric storm forecasting technique by artificial neural network , 2003 .

[33]  Haris Haralambous,et al.  Evolutionary Prediction of Total Electron Content over Cyprus , 2010, AIAI.

[34]  Vladimir Vovk,et al.  Ridge Regression Confidence Machine , 2001, International Conference on Machine Learning.

[35]  A. J. Gammerman,et al.  Plant promoter prediction with confidence estimation , 2005, Nucleic acids research.

[36]  H. Lundstedt,et al.  Prediction of geomagnetic storms from solar wind data using Elman Recurrent Neural Networks , 1996 .

[37]  Shuyuan Yang,et al.  Radar target recognition using contourlet packet transform and neural network approach , 2009, Signal Process..

[38]  Harris Papadopoulos,et al.  Qualified Prediction for Large Data Sets in the Case of Pattern Recognition , 2002, International Conference on Machine Learning and Applications.

[39]  Paula Fritzsche.  Tools in Artificial Intelligence , 2008 .

[40]  Alexander Gammerman,et al.  Pattern Recognition and Density Estimation under the General i.i.d. Assumption , 2001, COLT/EuroCOLT.

[41]  Konstantinos Koutroumbas,et al.  Time series autoregression technique implemented on-line in DIAS system for ionospheric forecast over Europe , 2008 .

[42]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[43]  Ersin Tulunay,et al.  Near-Earth space plasma modelling and forecasting , 2009 .

[44]  Alexander Gammerman,et al.  Serum Proteomic Abnormality Predating Screen Detection of Ovarian Cancer , 2009, Comput. J..

[45]  V. Deved,et al.  Proceedings of the 24th IASTED international conference on Artificial intelligence and applications , 2006 .

[46]  Alexander Gammerman,et al.  Qualified predictions for microarray and proteomics pattern diagnostics with confidence machines , 2005, Int. J. Neural Syst..