A Comparative Study of Methods for Collective Prediction Determination Using Interval Estimates

Recently, research on the Wisdom of Crowd (WoC) has been widely expanded by supporting interval values as an additional representation of underlying predictions. Accordingly, instead of giving single values, ones can express their predictions on a given cognition problem in the form of interval values1. For such a representation, many methods have been proposed for aggregating underlying predictions based on their midpoints. In this case, of course, the outputs of the proposed methods are single values. In some situations, however, the aggregated prediction in the form of interval value can be better representation of underlying predictions. In the current study, we present a comparison of the use of different approaches for aggregating individual predictions including Interval Aggregation and MidPoint Aggregation. Experimental studies have been conducted to determine how do different aggregation methods influence the quality of the obtained collective prediction.

[1]  Thomas Langer,et al.  True Overconfidence in Interval Estimates: Evidence Based on a New Measure of Miscalibration , 2012 .

[2]  Ofer Arazy,et al.  Forecasting rain events - Meteorological models or collective intelligence? , 2015 .

[3]  Ngoc Thanh Nguyen,et al.  Advanced Methods for Inconsistent Knowledge Management , 2007, Advanced Information and Knowledge Processing.

[4]  David V. Budescu,et al.  Aggregating multiple probability intervals to improve calibration , 2015, Judgment and Decision Making.

[5]  Jens Krause,et al.  Detection Accuracy of Collective Intelligence Assessments for Skin Cancer Diagnosis. , 2015, JAMA dermatology.

[6]  Jeffrey A. Baars,et al.  Performance of national weather service forecasts compared to operational, consensus, and weighted model output statistics , 2005 .

[7]  Ngoc Thanh Nguyen Advanced Methods for Inconsistent Knowledge Management (Advanced Information and Knowledge Processing) , 2007 .

[8]  Stefan M. Herzog,et al.  Boosting medical diagnostics by pooling independent judgments , 2016, Proceedings of the National Academy of Sciences.

[9]  Joshua Klayman,et al.  Overconfidence in interval estimates. , 2004, Journal of experimental psychology. Learning, memory, and cognition.

[10]  Ngoc Thanh Nguyen Consensus-based Timestamps in Distributed Temporal Databases , 2001, Comput. J..

[11]  Ngoc Thanh Nguyen,et al.  Processing Inconsistency of Knowledge on Semantic Level , 2005, J. Univers. Comput. Sci..

[12]  O. Hinz,et al.  Are crowds on the internet wiser than experts? The case of a stock prediction community , 2014, Journal of Business Economics.

[13]  Fiona Fidler,et al.  Reducing Overconfidence in the Interval Judgments of Experts , 2010, Risk analysis : an official publication of the Society for Risk Analysis.

[14]  Michal Zgrzywa,et al.  Consensus Determining with Dependencies of Attributes with Interval Values , 2007, J. Univers. Comput. Sci..

[15]  Michael Vitale,et al.  The Wisdom of Crowds , 2015, Cell.

[16]  R. Clemen Combining forecasts: A review and annotated bibliography , 1989 .

[17]  M. Burgman,et al.  Collective wisdom: Methods of confidence interval aggregation , 2015 .

[18]  Thomas Langer,et al.  True Overconfidence in Interval Estimates: Evidence Based on a New Measure of Miscalibration: True Overconfidence in Interval Estimates , 2013 .

[19]  Neeraj Bharadwaj,et al.  How crowdsourcing improves prediction of market-oriented outcomes , 2016 .