Aggregating predictions from experts: A review of statistical methods, experiments, and applications

[1]  Mario P. Brito,et al.  A Behavioral Probabilistic Risk Assessment Framework for Managing Autonomous Underwater Vehicle Deployments , 2012 .

[2]  Roberto Casarin,et al.  Bayesian Nonparametric Calibration and Combination of Predictive Distributions , 2015, 1502.07246.

[3]  Georgios Paliouras,et al.  Stacking Classifiers for Anti-Spam Filtering of E-Mail , 2001, EMNLP.

[4]  Ian H. Witten,et al.  Issues in Stacked Generalization , 2011, J. Artif. Intell. Res..

[5]  P. Baldwin.  Weighting Components of a Composite Score Using Naïve Expert Judgments About Their Relative Importance , 2015, Applied psychological measurement.

[6]  Robert L. Winkler,et al.  The Consensus of Subjective Probability Distributions , 1968 .

[7]  A. Diederich,et al.  Evaluating and Combining Subjective Probability Estimates , 1997 .

[8]  Guoliang Li,et al.  Human-in-the-loop Data Integration , 2017, Proc. VLDB Endow..

[9]  Philippe Baecke,et al.  Investigating the added value of integrating human judgement into statistical demand forecasting systems , 2017 .

[10]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[11]  Sebastian Lerch,et al.  Combining predictive distributions for the statistical post-processing of ensemble forecasts , 2016, International Journal of Forecasting.

[12]  A. Graefe.  Predicting elections: Experts, polls, and fundamentals , 2018, Judgment and Decision Making.

[13]  A. H. Murphy,et al.  Hailfinder: A Bayesian system for forecasting severe weather , 1996 .

[14]  Peter A. Morris,et al.  Decision Analysis Expert Use , 1974 .

[15]  Simon French,et al.  Aggregating expert judgement , 2011 .

[16]  Erim Kardes,et al.  Calibration, sharpness and the weighting of experts in a linear opinion pool , 2015, Ann. Oper. Res..

[17]  R. L. Winkler,et al.  Averages of Forecasts: Some Empirical Results , 1983 .

[18]  C. Powell.  The Delphi technique: myths and realities. , 2003, Journal of advanced nursing.

[19]  J. Scott Armstrong,et al.  JUDGMENTAL BOOTSTRAPPING: INFERRING EXPERTS' RULES FOR FORECASTING , 2001 .

[20]  Roger Cooke,et al.  Validating Expert Judgment with the Classical Model , 2014 .

[21]  Tara G. Martin,et al.  Understanding and predicting the combined effects of climate change and land‐use change on freshwater macroinvertebrates and fish , 2014 .

[22]  Kaplan,et al.  ‘Combining Probability Distributions from Experts in Risk Analysis’ , 2000, Risk analysis : an official publication of the Society for Risk Analysis.

[23]  Samuel DiGangi,et al.  Assessing unidimensionality: A comparison of Rasch modeling, Parallel analysis, and TETRAD , 2007 .

[24]  Andreas Graefe,et al.  Accuracy gains of adding vote expectation surveys to a combined forecast of US presidential election outcomes , 2015 .

[25]  F. Paas,et al.  Recent advances in expertise research , 2005 .

[26]  M. G. Morgan.  Use (and abuse) of expert elicitation in support of decision making for public policy , 2014, Proceedings of the National Academy of Sciences.

[27]  David H. Wolpert,et al.  Stacked generalization , 1992, Neural Networks.

[28]  Rakesh K. Sarin,et al.  From the Editor - Median Aggregation, Scoring Rules, Expert Forecasts, Choices with Binary Attributes, Portfolio with Dependent Projects, and Information Security , 2013, Decis. Anal..

[29]  R. L. Winkler,et al.  Combining Economic Forecasts , 1986 .

[30]  C. Granger,et al.  Improved methods of combining forecasts , 1984 .

[31]  Jin Wei-liang.  Research progress on the durability design and life prediction of concrete structures , 2007 .

[32]  Yan-ju Liu,et al.  Preliminary Strategic Environmental Assessment of the Great Western Development Strategy: Safeguarding Ecological Security for a New Western China , 2011, Environmental Management.

[33]  Christian Genest,et al.  Combining Probability Distributions: A Critique and an Annotated Bibliography , 1986 .

[34]  Enrico Zio,et al.  Accounting for expert-to-expert variability: A potential source of bias in performance assessments of high-level radioactive waste repositories , 1997 .

[35]  Robin M. Hogarth,et al.  Cognitive Processes and the Assessment of Subjective Probability Distributions , 1975 .

[36]  Allègre L. Hadida,et al.  On the relative importance of linear model and human judge(s) in combined forecasting , 2013 .

[37]  Shouyang Wang,et al.  Improving Forecasting Performance by Exploiting Expert Knowledge: Evidence from Guangzhou Port , 2016, Int. J. Inf. Technol. Decis. Mak..

[38]  Roy C. Amara,et al.  Some views on the use of expert judgment , 1971 .

[39]  Derek W. Bunn.  The Synthesis of Predictive Models in Marketing Research , 1979 .

[40]  J S Evans,et al.  Use of probabilistic expert judgment in uncertainty analysis of carcinogenic potency. , 1994, Regulatory toxicology and pharmacology : RTP.

[41]  Gwyn Griffiths,et al.  A Bayesian approach for predicting risk of autonomous underwater vehicle loss during their missions , 2016, Reliab. Eng. Syst. Saf..

[42]  Jian Ma,et al.  A comparative assessment of ensemble learning for credit scoring , 2011, Expert Syst. Appl..

[43]  Michael Goldstein,et al.  Bayesian Forecasting for Complex Systems Using Computer Simulators , 2001 .

[44]  Keith J. Burnham,et al.  Fuzzy decision support system for demand forecasting with a learning mechanism , 2006, Fuzzy Sets Syst..

[45]  Robert T. Clemen,et al.  Comment on Cooke's classical method , 2008, Reliab. Eng. Syst. Saf..

[46]  R. L. Winkler,et al.  Coherent combination of experts' opinions , 1995 .

[47]  Lyle H. Ungar,et al.  The Good Judgment Project: A Large Scale Test of Different Methods of Combining Expert Predictions , 2012, AAAI Fall Symposium: Machine Aggregation of Human Judgment.

[48]  Sarah Wordsworth,et al.  Eliciting expert opinion for economic models: an applied example. , 2007, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[49]  John R. Wilson,et al.  The nature of expertise: a review. , 2006, Applied ergonomics.

[50]  Marc Vuillet,et al.  Uncertainty and Expert Assessment for Supporting Evaluation of Levees Safety , 2016 .

[51]  Julio Saez-Rodriguez,et al.  Crowdsourcing Network Inference: The DREAM Predictive Signaling Network Challenge , 2011, Science Signaling.

[52]  F. Johnson,et al.  Making do with less: must sparse data preclude informed harvest strategies for European waterbirds? , 2018, Ecological applications : a publication of the Ecological Society of America.

[53]  R. L. Winkler.  Combining Probability Distributions from Dependent Information Sources , 1981 .

[54]  L. E. Peterson,et al.  Eliciting expert opinion using the Delphi technique: identifying performance indicators for cardiovascular disease. , 1998, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[55]  George K. Karagiannidis,et al.  Efficient Machine Learning for Big Data: A Review , 2015, Big Data Res..

[56]  Khaled Rasheed,et al.  Decision tree and ensemble learning algorithms with their applications in bioinformatics. , 2011, Advances in experimental medicine and biology.

[57]  M. Burgman,et al.  The Value of Performance Weights and Discussion in Aggregated Expert Judgments , 2018, Risk analysis : an official publication of the Society for Risk Analysis.

[58]  Forecast Combination With Entry and Exit of Experts , 2009 .

[59]  M. Degroot.  A Bayesian view of assessing uncertainty and comparing expert opinion , 1988 .

[60]  Yael Grushka-Cockayne,et al.  Probability Forecasts and Their Combination: A Research Perspective , 2018, Decis. Anal..

[61]  Fabio Wasserfallen,et al.  Learning and the diffusion of regime contention in the Arab Spring , 2015 .

[62]  R. H. Edmundson.  Decomposition; a strategy for judgemental forecasting , 1990 .

[63]  Jonathan Baron,et al.  Two Reasons to Make Aggregated Probability Forecasts More Extreme , 2014, Decis. Anal..

[64]  Kenneth F. Wallis,et al.  Combining forecasts – forty years later , 2011 .

[65]  Andreas Holzinger,et al.  Interactive machine learning for health informatics: when do we need the human-in-the-loop? , 2016, Brain Informatics.

[66]  Robin M. Hogarth,et al.  Judgmental Versus Statistical Prediction: Information Asymmetry and Combination Rules , 1993 .

[67]  Adrian E. Raftery,et al.  Combining Spatial Statistical and Ensemble Information in Probabilistic Weather Forecasts , 2007 .

[68]  Brett Houlding,et al.  Deriving the probability of a linear opinion pooling method being superior to a set of alternatives , 2017, Reliab. Eng. Syst. Saf..

[69]  Roger M Cooke,et al.  Out-of-sample validation for structured expert judgment of Asian carp establishment in Lake Erie , 2014, Integrated environmental assessment and management.

[70]  Bob Glahn,et al.  MOS Uncertainty Estimates in an Ensemble Framework , 2009 .

[71]  Stephen C. Hora,et al.  Median Aggregation of Distribution Functions , 2013, Decis. Anal..

[72]  William Remus,et al.  Judgemental forecasting in times of change , 1993 .

[73]  Graham T. F. Horn,et al.  Using Expert Judgment and Stakeholder Values to Evaluate Adaptive Management Options , 2004 .

[74]  Thomas L. Saaty,et al.  Expert System for Ice Hockey Game Prediction: Data Mining with Human Judgment , 2016, Int. J. Inf. Technol. Decis. Mak..

[75]  Marcus O'Connor,et al.  Exploring judgemental forecasting , 1992 .

[76]  Robert L. Winkler,et al.  Probabilistic Prediction: Some Experimental Results , 1971 .

[77]  Philip Hans Franses,et al.  Averaging Model Forecasts and Expert Forecasts: Why Does It Work? , 2011, Interfaces.

[78]  Changjun Liu,et al.  On a Simple and Efficient Approach to Probability Distribution Function Aggregation , 2017, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[79]  John C. Alexander.  Refining the Degree of Earnings Surprise: A Comparison of Statistical and Analysts' Forecasts , 1995 .

[80]  Gary B. Wills,et al.  Application of Bagging, Boosting and Stacking to Intrusion Detection , 2012, MLDM.

[81]  R. L. Winkler,et al.  Advances in Decision Analysis: Aggregating Probability Distributions , 2007 .

[82]  Sydney E. Scott,et al.  Psychological Strategies for Winning a Geopolitical Forecasting Tournament , 2014, Psychological science.

[83]  Nigel Harvey,et al.  Combining forecasts: What information do judges need to outperform the simple average? , 1999 .

[84]  Hajo Hippner,et al.  Crowdsourcing , 2012, Business & Information Systems Engineering.

[85]  Adrian E. Raftery,et al.  Locally Calibrated Probabilistic Temperature Forecasting Using Geostatistical Model Averaging and Local Bayesian Model Averaging , 2011 .

[86]  Enrico Zio,et al.  On the use of the analytic hierarchy process in the aggregation of expert judgments , 1996 .

[87]  R. Cooke.  Elicitation of expert opinions for uncertainty and risks , 2003 .

[88]  Javier M. Moguerza,et al.  Combination and Selection of Traffic Safety Expert Judgments for the Prevention of Driving Risks , 2012, Sensors.

[89]  B Kleinmuntz,et al.  Why we still use our heads instead of formulas: toward an integrative approach. , 1990, Psychological bulletin.

[90]  A. H. Murphy,et al.  Probability Forecasts: A Survey of National Weather Service Forecasters , 1974 .

[91]  F. Galton.  Vox Populi , 1907, Nature.

[92]  J. M. Bates,et al.  The Combination of Forecasts , 1969 .

[93]  K. Mengersen,et al.  Eliciting Expert Knowledge in Conservation Science , 2012, Conservation biology : the journal of the Society for Conservation Biology.

[94]  Haiyan Song,et al.  Combining statistical and judgmental forecasts via a web-based tourism demand forecasting system , 2013 .

[95]  Jeremy E. Oakley,et al.  Uncertain Judgements: Eliciting Experts' Probabilities , 2006 .

[96]  Ray Adams,et al.  An Acceptability Predictor for Websites , 2009, HCI.

[97]  K. A. Ericsson,et al.  Science Current Directions in Psychological of Expert and Exceptional Performance Capturing the Naturally Occurring Superior Performance of Experts in the Laboratory : toward a Science on Behalf Of: Association for Psychological Science , 2022 .

[98]  Tung Bui,et al.  Aggregating and updating experts' knowledge: An experimental evaluation of five classification techniques , 1996 .

[99]  R. Cooke.  The Aggregation of Expert Judgment: Do Good Things Come to Those Who Weight? , 2015, Risk analysis : an official publication of the Society for Risk Analysis.

[100]  Roger Cooke,et al.  Calibration and information in expert resolution; a classical approach , 1988, Autom..

[101]  Mark T. Spence,et al.  The moderating effects of problem characteristics on experts' and novices' judgments , 1997 .

[102]  D. Bunn,et al.  Interaction of judgemental and statistical forecasting methods: issues & , 1991 .

[103]  E. L. Kelly.  Clinical versus statistical prediction: A theoretical analysis and review of the evidence. , 1955 .

[104]  Y Al-JarrahOmar,et al.  Efficient Machine Learning for Big Data , 2015 .

[105]  J. Armstrong,et al.  PRINCIPLES OF FORECASTING 1 Principles of Forecasting : A Handbook for Researchers and Practitioners , 2006 .

[106]  C. Lintott,et al.  Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey , 2008, 0804.4483.

[107]  D. Bunn,et al.  Statistical efficiency in the linear combination of forecasts , 1985 .

[108]  Daniel M. Tartakovsky,et al.  Probabilistic risk analysis in subsurface hydrology , 2007 .

[109]  A. H. Murphy,et al.  subjective probability forecasting experiments in meteorology: some preliminary results , 1974 .

[110]  Alfred G. Cuzán,et al.  Accuracy of Combined Forecasts for the 2012 Presidential Election: The PollyVote , 2014, PS: Political Science & Politics.

[111]  Peter Willett,et al.  The Porter stemming algorithm: then and now , 2006, Program.

[112]  Frank Elberzhager,et al.  Support planning and controlling of early quality assurance by combining expert judgment and defect data—a case study , 2010, Empirical Software Engineering.

[113]  Gavin Brown,et al.  Ensemble Learning , 2010, Encyclopedia of Machine Learning and Data Mining.

[114]  Jack T. Dennerlein,et al.  Expertise, credibility of system forecasts and integration methods in judgmental demand forecasting , 2017 .

[115]  Juha M. Alho.  Estimating the strength of expert judgement: The case of US mortality forecasts , 1992 .

[116]  Xiaofeng Wang,et al.  Probabilistic inversion techniques in quantitative risk assessment for power system load forecasting , 2008, 2008 International Conference on Information and Automation.

[117]  A. Dawid.  The geometry of proper scoring rules , 2007 .

[118]  Yinda Zhang,et al.  LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop , 2015, ArXiv.

[119]  Rajkumar Buyya,et al.  The anatomy of big data computing , 2015, Softw. Pract. Exp..

[120]  M. Stone.  The Opinion Pool , 1961 .

[121]  T. Gneiting,et al.  Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules , 2011 .

[122]  Ming-Hui Chen,et al.  Statistical methods and computing for big data. , 2015, Statistics and its interface.

[123]  P. Goodwin,et al.  Judgmental forecasting: A review of progress over the last 25 years , 2006 .

[124]  Muhammad Imran Yousuf,et al.  Using Experts` Opinions Through Delphi Technique , 2007 .

[125]  Mark E. Borsuk,et al.  Predictive Assessment of Fish Health and Fish Kills in the Neuse River Estuary Using Elicited Expert Judgment , 2004 .

[126]  Dorota Kurowicka,et al.  Probabilistic Inversion in Priority Setting of Emerging Zoonoses , 2010, Risk analysis : an official publication of the Society for Risk Analysis.

[127]  T. Gneiting,et al.  Combining Predictive Distributions , 2011, 1106.1638.

[128]  O. M. Nápoles,et al.  Delft University of Technology Characterization of Precipitation through Copulas and Expert Judgement for Risk Assessment of Infrastructure , 2017 .

[129]  Reid Priedhorsky,et al.  Epidemic Forecasting is Messier Than Weather Forecasting: The Role of Human Behavior and Internet Data Streams in Epidemic Forecast. , 2016, The Journal of infectious diseases.

[130]  J. Armstrong,et al.  Relative Accuracy of Judgmental and Extrapolative Methods in Forecasting Annual Earnings , 1983 .

[131]  Sutapa Pramanik,et al.  Interval type-2 fuzzy logic and its application to occupational safety risk performance in industries , 2019, Soft Comput..

[132]  William J. Hurley,et al.  Combining expert judgment: On the performance of trimmed mean vote aggregation procedures in the presence of strategic voting , 2002, Eur. J. Oper. Res..

[133]  Changyong Lee,et al.  Robust Future‐Oriented Technology Portfolios: Black–Litterman Approach , 2013 .

[134]  Andreas Graefe,et al.  Combining Forecasts: An Application to Elections , 2013 .

[135]  R. Keeney.  A Group Preference Axiomatization with Cardinal Utility , 1976 .

[136]  T. Gneiting,et al.  Combining probability forecasts , 2010 .

[137]  Xueqing Zhang,et al.  Bayesian Analytics for Estimating Risk Probability in PPP Waste-to-Energy Projects , 2018, Journal of Management in Engineering.

[138]  A. Timmermann,et al.  Combining expert forecasts: Can anything beat the simple average? , 2013 .

[139]  R. Clemen.  Combining forecasts: A review and annotated bibliography , 1989 .

[140]  Bernard Zenko,et al.  Is Combining Classifiers with Stacking Better than Selecting the Best One? , 2004, Machine Learning.

[141]  M. J. van der Laan,et al.  Statistical Applications in Genetics and Molecular Biology Super Learner , 2010 .

[142]  Lyle H. Ungar,et al.  Probability aggregation in time-series: Dynamic hierarchical modeling of sparse expert beliefs , 2014, 1408.0087.