Combining forecasts: What information do judges need to outperform the simple average?

Abstract Previous work has shown that combinations of separate forecasts produced by judgment are inferior to those produced by simple averaging. However, in that research judges were not informed of outcomes after producing each combined forecast. Our first experiment shows that when they are given this information, they learn to weight the separate forecasts appropriately. However, their judgments, though improved, are still not significantly better than the simple average because they contain a random error component. Bootstrapping can be used to remove this inconsistency and produce results that outperform the average. In our second and third experiments, we provided judges with information about errors made by the individual forecasters. Results show that providing information about their mean absolute percentage errors updated each period enables judges to combine their forecasts in a way that outperforms the simple average.

[1]  Fred L. Collopy,et al.  Error Measures for Generalizing About Forecasting Methods: Empirical Comparisons , 1992 .

[2]  M. Björkman.  FEEDFORWARD AND FEEDBACK AS DETERMINERS OF KNOWLEDGE AND POLICY: NOTES ON A NEGLECTED ISSUE , 1972 .

[3]  Alexander J. Wearing,et al.  Feedback and the forecasting of exponential change , 1991 .

[4]  Larry King,et al.  Feedback and task predictability as determinants of performance in multiple cue probability learning tasks , 1976 .

[5]  L. Phillips.  A theory of requisite decision models , 1984 .

[6]  Lewis R. Goldberg,et al.  Man versus model of man: A rationale, plus some evidence, for a method of improving on clinical inferences. , 1970 .

[7]  M. Lindell.  Cognitive and Outcome Feedback in Multiple-Cue Probability Learning Tasks. , 1976 .

[8]  R. H. Edmundson,et al.  The accuracy of combining judgemental and statistical forecasts , 1986 .

[9]  Nada R. Sanders,et al.  The impact of task properties feedback on time series judgmental forecasting tasks , 1997 .

[10]  Leslie B. Hammer,et al.  Task information, cognitive information, or functional validity information: Which components of cognitive feedback affect performance?☆ , 1992 .

[11]  Earl A. Alluisi,et al.  Principles of skill acquisition , 1969 .

[12]  B. Tabachnick,et al.  Using Multivariate Statistics , 1983 .

[13]  Robert H. Ashton,et al.  Aggregating Subjective Forecasts: Some Empirical Results , 1985 .

[14]  J. M. Bates,et al.  The Combination of Forecasts , 1969 .

[15]  Neal Schmitt,et al.  Types of task information feedback in multiple-cue probability learning. , 1977 .

[16]  M. O'Connor,et al.  Judgemental and statistical time series forecasting: a review of the literature , 1996 .

[17]  K R Hammond,et al.  Differential feedback in two multiple-cue probability learning tasks. , 1965, Behavioral science.

[18]  Laureen A. Maines.  An experimental examination of subjective forecast combination , 1996 .

[19]  Paul Goodwin,et al.  Heuristics, biases and improvement strategies in judgmental time series forecasting , 1994 .

[20]  N. Wiggins,et al.  Man versus model of man revisited: The forecasting of graduate school success. , 1971 .

[21]  Pam Angus-Leppan,et al.  The Forecasting Accuracy of Trainee Accountants Using Judgemental and Statistical Techniques , 1986 .

[22]  B. Brehmer.  In one word: Not from experience. , 1980 .

[23]  M. Doherty,et al.  Effects of cognitive feedback on performance. , 1989 .

[24]  Marcus O'Connor,et al.  Judgement or models: The importance of task differences , 1996 .

[25]  Robert L. Winkler,et al.  The accuracy of extrapolation (time series) methods: Results of a forecasting competition , 1982 .

[26]  Kenneth R. Hammond,et al.  Quasi-rationality, quarrels and new conceptions of feedback. , 1971 .

[27]  Donald H. Deane,et al.  Acquisition and application of knowledge in complex inference tasks. , 1972 .

[28]  M. Doherty,et al.  Chapter 5 Cognitive Feedback , 1988 .

[29]  Kenneth R. Hammond,et al.  Negative effects of outcome-feedback in multiple-cue probability learning. , 1973 .

[30]  C. Granger.  Invited review combining forecasts—twenty years later , 1989 .

[31]  Henry L. Tosi.  A Theory of Goal Setting and Task Performance , 1991 .

[32]  M. O'Connor,et al.  Judgemental adjustment of initial forecasts: Its effectiveness and biases , 1995 .

[33]  K R Hammond,et al.  Computer Graphics as an Aid to Learning , 1971, Science.

[34]  Allen Parducci,et al.  Response Bias and Contextual Effects: When Biased? , 1990 .

[35]  L. Adelman.  The influence of formal, substantive, and contextual task properties on the relative effectiveness of different forms of feedback in multiple-cue probability learning tasks☆ , 1981 .

[36]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[37]  Timothy D. Lee,et al.  Motor Control and Learning: A Behavioral Emphasis , 1982 .

[38]  R. Dawes,et al.  Linear models in decision making. , 1974 .

[39]  R. Clemen.  Combining forecasts: A review and annotated bibliography , 1989 .

[40]  William Remus,et al.  Does Feedback Improve the Accuracy of Recurrent Judgmental Forecasts , 1996 .

[41]  Marcus O'Connor,et al.  An examination of the accuracy of judgmental extrapolation of time series , 1985 .

[42]  Ray W. Cooksey,et al.  Judgment analysis : theory, methods, and applications , 1996 .

[43]  N. Harvey,et al.  Taking Advice: Accepting Help, Improving Judgment, and Sharing Responsibility☆☆☆ , 1997 .

[44]  C. Granger,et al.  Improved methods of combining forecasts , 1984 .