Methods For Combining Experts' Probability Assessments

This article reviews statistical techniques for combining multiple probability distributions. The framework is that of a decision maker who consults several experts regarding some events. The experts express their opinions in the form of probability distributions. The decision maker must aggregate the experts' distributions into a single distribution that can be used for decision making. Two classes of aggregation methods are reviewed. When using a supra Bayesian procedure, the decision maker treats the expert opinions as data that may be combined with its own prior distribution via Bayes' rule. When using a linear opinion pool, the decision maker forms a linear combination of the expert opinions. The major feature that makes the aggregation of expert opinions difficult is the high correlation or dependence that typically occurs among these opinions. A theme of this paper is the need for training procedures that result in experts with relatively independent opinions or for aggregation methods that implicitly or explicitly model the dependence among the experts. Analyses are presented that show that m dependent experts are worth the same as k independent experts where k m. In some cases, an exact value for k can be given; in other cases, lower and upper bounds can be placed on k.

[1]  G. E. Peterson,et al.  Control Methods Used in a Study of the Vowels , 1951 .

[2]  E. H. Shuford,et al.  Admissible probability measurement procedures , 1966, Psychometrika.

[3]  R. L. Winkler.  Scoring Rules and the Evaluation of Probability Assessors , 1969 .

[4]  J. M. Bates,et al.  The Combination of Forecasts , 1969 .

[5]  J. Dickinson.  Some Statistical Results in the Combination of Forecasts , 1973 .

[6]  Peter A. Morris,et al.  Decision Analysis Expert Use , 1974 .

[7]  J. Dickinson.  Some Comments on the Combination of Forecasts , 1975 .

[8]  C. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[9]  A. Tversky,et al.  On the Reconciliation of Probability Assessments , 1979 .

[10]  Simon French,et al.  Updating of Belief in the Light of Someone Else's Opinion , 1980 .

[11]  K. McConway.  Marginalization and Linear Opinion Pools , 1981 .

[12]  R. L. Winkler.  Combining Probability Distributions from Dependent Information Sources , 1981 .

[13]  R. Bordley.  The Combination of Forecasts: a Bayesian Approach , 1982 .

[14]  David Lindley.  The Improvement of Probability Judgements , 1982 .

[15]  M. Degroot,et al.  Comparing Probability Forecasters: Basic Binary Concepts and Multivariate Extensions , 1983 .

[16]  C. Granger,et al.  Improved methods of combining forecasts , 1984 .

[17]  Robert L. Winkler,et al.  Limits for the Precision and Value of Information from Dependent Sources , 1985, Oper. Res..

[18]  C. E. Agnew.  Multiple Probability Assessments by Dependent Experts , 1985 .

[19]  G. Sperling,et al.  Tradeoffs between stereopsis and proximity luminance covariance as determinants of perceived 3D structure , 1986, Vision Research.

[20]  R. Clemen.  Linear constraints and the efficiency of combined forecasts , 1986 .

[21]  Christian Genest,et al.  Combining Probability Distributions: A Critique and an Annotated Bibliography , 1986 .

[22]  R. Bordley.  Linear combination of forecasts with an intercept: A bayesian approach , 1986 .

[23]  C. Genest,et al.  Further evidence against independence preservation in expert judgement synthesis , 1987 .

[24]  N. Graham.  Visual Pattern Analyzers , 1989 .

[25]  Christian Genest,et al.  Allocating the weights in the linear opinion pool , 1990 .

[26]  Roger M. Cooke,et al.  Statistics in Expert Resolution: A Theory of Weights for Combining Expert Opinion , 1990 .

[27]  James J. Clark,et al.  Data Fusion for Sensory Information Processing Systems , 1990 .

[28]  K Nakayama,et al.  Toward a neural understanding of visual surface representation. , 1990, Cold Spring Harbor symposia on quantitative biology.

[29]  Geoffrey E. Hinton,et al.  Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[30]  Harris Drucker,et al.  Improving Performance in Neural Networks Using a Boosting Algorithm , 1992, NIPS.

[31]  David H. Wolpert,et al.  Stacked generalization , 1992, Neural Networks.

[32]  Mongi A. Abidi,et al.  Data fusion in robotics and machine intelligence , 1992 .

[33]  Adam Krzyżak,et al.  Methods of combining multiple classifiers and their applications to handwriting recognition , 1992, IEEE Trans. Syst. Man Cybern..

[34]  M. Landy,et al.  A perturbation analysis of depth perception from combinations of texture and motion cues , 1993, Vision Research.

[35]  M. Perrone.  Improving regression estimation: Averaging methods for variance reduction with extensions to general convex measure optimization , 1993 .

[36]  John C. Trueswell,et al.  Surface segmentation mechanisms and motion perception , 1993, Vision Research.

[37]  Robert A. Jacobs,et al.  Hierarchical Mixtures of Experts and the EM Algorithm , 1993, Neural Computation.

[38]  Michael I. Jordan,et al.  Hierarchical Mixtures of Experts and the EM Algorithm , 1994, Neural Computation.

[39]  E. Marg.  A VISION OF THE BRAIN , 1994 .

[40]  Michael I. Jordan,et al.  Hierarchical Mixtures of Experts and the EM Algorithm , 1994 .

[41]  A. Gelfand,et al.  Modeling Expert Opinion Arising as a Partial Probabilistic Specification , 1995 .

[42]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[43]  Ronny Meir,et al.  Bias, variance and the combination of estimators; The case of linear least squares , 1995 .

[44]  Johannes R. Sveinsson,et al.  Hybrid consensus theoretic classification , 1996, IGARSS '96. 1996 International Geoscience and Remote Sensing Symposium.

[45]  R. Tibshirani,et al.  Combining Estimates in Regression and Classification , 1996 .

[46]  Sherif Hashem,et al.  Optimal Linear Combinations of Neural Networks , 1997, Neural Networks.

[47]  Volker Tresp,et al.  Averaging Regularized Estimators , 1997, Neural Computation.

[48]  Ah Chung Tsoi,et al.  Face recognition: a convolutional neural-network approach , 1997, IEEE Trans. Neural Networks.

[49]  W.J. Tompkins,et al.  A patient-adaptable ECG beat classifier using a mixture of experts approach , 1997, IEEE Transactions on Biomedical Engineering.

[50]  Johannes R. Sveinsson,et al.  Parallel consensual neural networks , 1997, IEEE Trans. Neural Networks.

[51]  Kuldip K. Paliwal,et al.  Bidirectional recurrent neural networks , 1997, IEEE Trans. Signal Process..