Meta-Analysis of the Effect of Natural Frequencies on Bayesian Reasoning

The natural frequency facilitation effect describes the finding that people are better able to solve descriptive Bayesian inference tasks when represented as joint frequencies obtained through natural sampling, known as natural frequencies, than as conditional probabilities. The present meta-analysis reviews 20 years of research seeking to address when, why, and for whom natural frequency formats are most effective. We review contributions from research associated with the 2 dominant theoretical perspectives, the ecological rationality framework and nested-sets theory, and test potential moderators of the effect. A systematic review of relevant literature yielded 35 articles representing 226 performance estimates. These estimates were statistically integrated using a bivariate mixed-effects model that yields summary estimates of average performances across the 2 formats and estimates of the effects of different study characteristics on performance. These study characteristics range from moderators representing individual characteristics (e.g., numeracy, expertise), to methodological differences (e.g., use of incentives, scoring criteria) and features of problem representation (e.g., short menu format, visual aid). Short menu formats (less computationally complex representations showing joint-events) and visual aids demonstrated some of the strongest moderation effects, improving performance for both conditional probability and natural frequency formats. A number of methodological factors (e.g., exposure to both problem formats) were also found to affect performance rates, emphasizing the importance of a systematic approach. We suggest how research on Bayesian reasoning can be strengthened by broadening the definition of successful Bayesian reasoning to incorporate choice and process and by applying different research methodologies.

[1]  Lisa M. Schwartz,et al.  PSYCHOLOGICAL SCIENCE IN THE PUBLIC INTEREST Helping Doctors and Patients Make Sense of Health Statistics , 2022 .

[2]  Hendrik Friederichs,et al.  Using Tree Diagrams without Numerical Values in Addition to Relative Numbers Improves Students’ Numeracy Skills , 2014, Medical decision making : an international journal of the Society for Medical Decision Making.

[3]  J. Koehler.  The base rate fallacy reconsidered: Descriptive, normative, and methodological challenges , 1996, Behavioral and Brain Sciences.

[4]  A. Tversky,et al.  On the study of statistical intuitions , 1982, Cognition.

[5]  Miroslav Sirota,et al.  On Bayesian problem-solving: helping Bayesians solve simple Bayesian word problems , 2015, Front. Psychol..

[6]  F. Vallée-Tourangeau,et al.  How to train your Bayesian: a problem-representation transfer rather than a format-representation shift explains training effects. , 2015, Quarterly journal of experimental psychology.

[7]  Aidan Feeney,et al.  Whose statistical reasoning is facilitated by a causal structure intervention? , 2014, Psychonomic Bulletin & Review.

[8]  H C Van Houwelingen,et al.  A bivariate approach to meta-analysis. , 1993, Statistics in medicine.

[9]  Macchi,et al.  Partitive Formulation of Information in Probabilistic Problems: Beyond Heuristics and Frequency Format Explanations. , 2000, Organizational behavior and human decision processes.

[10]  B. Rimer,et al.  General Performance on a Numeracy Scale among Highly Educated Samples , 2001, Medical decision making : an international journal of the Society for Medical Decision Making.

[11]  L. Cosmides,et al.  Are humans good intuitive statisticians after all? Rethinking some conclusions from the literature on judgment under uncertainty , 1996, Cognition.

[12]  Eamonn Ferguson,et al.  Incentives, expertise, and medical decisions: testing the robustness of natural frequency framing. , 2013, Health psychology : official journal of the Division of Health Psychology, American Psychological Association.

[13]  Gaëlle Vallée-Tourangeau,et al.  Interactivity Fosters Bayesian Reasoning without Instruction [Data set and code book] , 2016 .

[14]  David A. Lagnado,et al.  Dual concerns with the dualist approach , 2007 .

[15]  Nick Chater,et al.  Bayesian models of cognition. , 2010, Wiley interdisciplinary reviews. Cognitive science.

[16]  K. Stanovich,et al.  Advancing the rationality debate , 2000, Behavioral and Brain Sciences.

[17]  Tevye R. Krynski,et al.  The role of causality in judgment under uncertainty. , 2007, Journal of experimental psychology. General.

[18]  Kimihiko Yamagishi,et al.  Facilitating normative judgments of conditional probability: frequency or nested sets? , 2003, Experimental psychology.

[19]  Gretchen B. Chapman,et al.  Retrospective Frequency Formats Promote Consistent Experience-Based Bayesian Judgments , 2012 .

[20]  T. Caraco,et al.  RISK AND FORAGING IN STOCHASTIC ENVIRONMENTS , 1986 .

[21]  Christopher P. Cerasoli,et al.  Intrinsic Motivation and Extrinsic Incentives Jointly Predict Performance , 2014 .

[22]  Gerd Gigerenzer,et al.  Children can solve Bayesian problems: the role of representation in mental computation , 2006, Cognition.

[23]  Jeffrey M. Stibel,et al.  Frequency illusions and other fallacies , 2003 .

[24]  Aidan Feeney,et al.  When does Information about Causal Structure Improve Statistical Reasoning? , 2014, Quarterly journal of experimental psychology.

[25]  Peter M. Todd,et al.  Building the Theory of Ecological Rationality , 2015, Minds and Machines.

[26]  J. Biernaskie,et al.  Bumblebees Learn to Forage like Bayesians , 2009, The American Naturalist.

[27]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[28]  I. White,et al.  Quantifying the impact of between-study heterogeneity in multivariate meta-analyses , 2012, Statistics in medicine.

[29]  G. Gigerenzer,et al.  Using icon arrays to communicate medical risks: overcoming low numeracy. , 2009, Health psychology : official journal of the Division of Health Psychology, American Psychological Association.

[30]  Gary L. Brase,et al.  Participant recruitment methods and statistical reasoning performance , 2006, Quarterly journal of experimental psychology.

[31]  I. Olkin,et al.  The case of the misleading funnel plot , 2006, BMJ : British Medical Journal.

[32]  David R. Mandel,et al.  Nested sets theory, full stop: Explaining performance on Bayesian inference tasks without dual-systems assumptions , 2007 .

[33]  Vittorio Girotto,et al.  Basic understanding of posterior probability , 2015, Front. Psychol..

[34]  P. Todd,et al.  Ecological Rationality: Intelligence in the World , 2012 .

[35]  P. Todd,et al.  Evolutionary Cognitive Psychology , 2015, Encyclopedia of Evolutionary Psychological Science.

[36]  G. Gigerenzer.  On Narrow Norms and Vague Heuristics: A Reply to Kahneman and Tversky (1996) , 1996 .

[37]  Peter Salmon,et al.  Health professionals' and service users' interpretation of screening test results: experimental study , 2006, BMJ : British Medical Journal.

[38]  Ulrich Hoffrage,et al.  Simplifying Bayesian Inference: The General Case , 1999 .

[39]  D. Kahneman,et al.  Heuristics and Biases: The Psychology of Intuitive Judgment , 2002 .

[40]  Lisa M. Schwartz,et al.  The Role of Numeracy in Understanding the Benefit of Screening Mammography , 1997, Annals of Internal Medicine.

[41]  Katharina Böcherer-Linder,et al.  The Impact of Visualizing Nested Sets. An Empirical Study on Tree Diagrams and Unit Squares , 2017, Front. Psychol..

[42]  Daniel Afergan,et al.  Improving Bayesian Reasoning: The Effects of Phrasing, Visualization, and Spatial Ability , 2016, IEEE Transactions on Visualization and Computer Graphics.

[43]  L. Hedges,et al.  Fixed- and random-effects models in meta-analysis. , 1998 .

[44]  Clarissa A. Thompson,et al.  An integrated theory of whole number and fractions development , 2011, Cognitive Psychology.

[45]  Wim De Neys,et al.  Evolutionary modules and Bayesian facilitation: The role of general cognitive resources , 2013 .

[46]  P. Todd,et al.  Environments That Make Us Smart , 2007 .

[47]  Simon Breslav,et al.  Benefits of visualization in the Mammography Problem , 2015, Int. J. Hum. Comput. Stud..

[48]  Barbara Tversky,et al.  Visualizing Thought , 2011, Top. Cogn. Sci..

[49]  Philip N. Johnson-Laird,et al.  Naive Probability: A Mental Model Theory of Extensional Reasoning , 1999 .

[50]  David Trafimow.  Why the empirical literature fails to support or disconfirm modular or dual-process models , 2007 .

[51]  Jonathan Evans,et al.  Frequency versus probability formats in statistical word problems , 2000, Cognition.

[52]  A. Tversky,et al.  Subjective Probability: A Judgment of Representativeness , 1972 .

[53]  F. Vallée-Tourangeau,et al.  Interactivity fosters Bayesian reasoning without instruction. , 2015, Journal of experimental psychology. General.

[54]  D. Sobel.  Running head: CHILDREN’S CAUSAL INFERENCES Children’s causal inferences from indirect evidence: Backwards blocking and Bayesian reasoning in preschoolers , 2003 .

[55]  Julian N. Marewski,et al.  Cognitive niches: an ecological model of strategy selection. , 2011, Psychological review.

[56]  Georg Bruckmaier,et al.  Effects of visualizing statistical information – an empirical study on tree diagrams and 2 × 2 tables , 2015, Front. Psychol..

[57]  Katarzyna Idzikowska,et al.  Bayesian probability estimates are not necessary to make choices satisfying Bayes’ rule in elementary situations , 2015, Front. Psychol..

[58]  Jeffrey R. Stevens,et al.  A statistical taxonomy and another “chance” for natural frequencies , 2007 .

[59]  Robbie C. M. van Aert,et al.  Meta-analysis using effect size distributions of only statistically significant studies. , 2015, Psychological methods.

[60]  Wolfgang Viechtbauer,et al.  Conducting Meta-Analyses in R with the metafor Package , 2010 .

[61]  R. Hertwig,et al.  The description–experience gap in risky choice , 2009, Trends in Cognitive Sciences.

[62]  Konheim Kalkstein,et al.  Facilitation of Bayesian decision making , 2008 .

[63]  L A Real,et al.  Animal choice behavior and the evolution of cognitive architecture , 1991, Science.

[64]  Gary L. Brase.  The power of representation and interpretation: Doubling statistical reasoning performance with icons and frequentist interpretations of ambiguous numbers , 2014 .

[65]  J. Neumann,et al.  Theory of games and economic behavior , 1945, 100 Years of Math Milestones.

[66]  A. Tversky,et al.  On the psychology of prediction , 1973 .

[67]  William P. Neace,et al.  Frequency formats, probability formats, or problem structure? A test of the nested-sets hypothesis in an extensional reasoning task , 2008 .

[68]  David J. Weiss,et al.  Conservatism in a Simple Probability Inference Task , 2008 .

[69]  Wolfgang Viechtbauer,et al.  Outlier and influence diagnostics for meta‐analysis , 2010, Research synthesis methods.

[70]  Simon McNair.  Beyond the status-quo: research on Bayesian reasoning must develop in both theory and method , 2015, Front. Psychol..

[71]  Juan Merlo,et al.  Natural frequencies improved diagnostic inference among medical students i Sweden , 2018, Lakartidningen.

[72]  David M. Sobel,et al.  Causal learning mechanisms in very young children: two-, three-, and four-year-olds infer causal relations from patterns of variation and covariation. , 2001, Developmental psychology.

[73]  G. Gigerenzer,et al.  Teaching Bayesian reasoning in less than two hours. , 2001, Journal of experimental psychology. General.

[74]  Jonathan D. Nelson,et al.  Asking Better Questions: How Presentation Formats Influence Information Search , 2017, Journal of experimental psychology. Learning, memory, and cognition.

[75]  S Duval,et al.  Trim and Fill: A Simple Funnel‐Plot–Based Method of Testing and Adjusting for Publication Bias in Meta‐Analysis , 2000, Biometrics.

[76]  U. Hoffrage,et al.  Toward an ecological analysis of Bayesian inferences: how task characteristics influence responses , 2015, Front. Psychol..

[77]  Miroslav Sirota,et al.  ROLE OF NUMERACY AND COGNITIVE REFLECTION IN BAYESIAN REASONING WITH NATURAL FREQUENCIES , 2011 .

[78]  Miroslav Sirota,et al.  Ecological rationality or nested sets? Individual differences in cognitive processing predict Bayesian reasoning , 2014, Psychonomic bulletin & review.

[79]  Ralph Hertwig,et al.  Nudge Versus Boost: How Coherent are Policy and Theory? , 2016, Minds and Machines.

[80]  Barbara Tversky,et al.  Spatial schemas in depictions , 1999 .

[81]  David M. Sobel,et al.  Domain generality and specificity in children's causal inference about ambiguous data. , 2009, Developmental psychology.

[82]  Gerd Gigerenzer,et al.  The recognition heuristic: A decade of research , 2011, Judgment and Decision Making.

[83]  L. Beach,et al.  Man as an Intuitive Statistician , 2022 .

[84]  Natalia Kochetova,et al.  Improving Auditors’ Fraud Judgments Using a Frequency Response Mode , 2010 .

[85]  M. Bar-Hillel.  The base-rate fallacy in probability judgments. , 1980 .

[86]  Jennifer Tsai,et al.  Interactive Visualizations to Improve Bayesian Reasoning , 2011 .

[87]  Gerd Gigerenzer,et al.  Why do frequency formats improve Bayesian reasoning? Cognitive algorithms work on information, which needs representation , 1996, Behavioral and Brain Sciences.

[88]  Hannes Rakoczy,et al.  Apes are intuitive statisticians , 2014, Cognition.

[89]  G. Gigerenzer,et al.  Representation facilitates reasoning: what natural frequencies are and what they are not , 2002, Cognition.

[90]  Ulrich Hoffrage,et al.  Natural frequencies facilitate diagnostic inferences of managers , 2015, Front. Psychol..

[91]  Klaus Fiedler,et al.  Heuristics and biases , 2015 .

[92]  Miroslav Sirota,et al.  Doctor, what does my positive test mean? From Bayesian textbook tasks to personalized risk communication , 2015, Front. Psychol..

[93]  W G Cole,et al.  Three Graphic Representations to Aid Bayesian Inference , 1988, Methods of Information in Medicine.

[94]  David R. Mandel,et al.  The psychology of Bayesian reasoning , 2014, Front. Psychol..

[95]  J. Henrich,et al.  The weirdest people in the world? , 2010, Behavioral and Brain Sciences.

[96]  Eric D. Johnson,et al.  Words, numbers, & numeracy: Diminishing individual differences in Bayesian reasoning , 2013 .

[97]  Gary L. Brase,et al.  Good fences make for good neighbors but bad science: a review of what improves Bayesian reasoning and why , 2015, Front. Psychol..

[98]  G. Gigerenzer.  Ecological intelligence: An adaptation for frequencies , 1997 .

[99]  Gernot D. Kleiter,et al.  Natural Sampling: Rationality without Base Rates , 1994 .

[100]  Christopher P. Cerasoli,et al.  Intrinsic motivation and extrinsic incentives jointly predict performance: a 40-year meta-analysis. , 2014, Psychological bulletin.

[101]  Elisabet Tubau,et al.  Comprehension and computation in Bayesian problem solving , 2015, Front. Psychol..

[102]  G. Kalyanaram,et al.  Nudge: Improving Decisions about Health, Wealth, and Happiness , 2011 .

[103]  Gerd Gigerenzer,et al.  On the Supposed Evidence for Libertarian Paternalism , 2015, Review of Philosophy and Psychology.

[104]  Giorgio Vallortigara,et al.  Probabilistic cognition in two indigenous Mayan groups , 2014, Proceedings of the National Academy of Sciences.

[105]  D. Altman,et al.  Measuring inconsistency in meta-analyses , 2003, BMJ : British Medical Journal.

[106]  A. Tversky,et al.  Extensional versus intuitive reasoning: the conjunction fallacy in probability judgment , 1983 .

[107]  Momme von Sydow,et al.  Heuristics and biases , 2015 .

[108]  Gorka Navarrete,et al.  Ecological Rationality and Evolution: The Mind Really Works that Way? , 2011, Front. Psychology.

[109]  Gerd Gigerenzer,et al.  How to Improve Bayesian Reasoning Without Instruction: Frequency Formats , 1995 .

[110]  Edward T. Cokely,et al.  Science Current Directions in Psychological , 2010 .

[111]  Gerd Gigerenzer,et al.  The role of representation in Bayesian reasoning: Correcting common misconceptions , 2007 .

[112]  Yuichi Amitani,et al.  The natural frequency hypothesis and evolutionary arguments , 2015 .

[113]  J. Corter,et al.  The Process of Probability Problem Solving: Use of External Visual Representations , 2010 .

[114]  Gerd Gigerenzer,et al.  Natural frequencies improve Bayesian reasoning in simple and complex inference tasks , 2015, Front. Psychol..

[115]  G Gigerenzer,et al.  Using natural frequencies to improve diagnostic inferences , 1998, Academic medicine : journal of the Association of American Medical Colleges.

[116]  John Tooby,et al.  W 2 Can a General Deontic Logic Capture the Facts of Human Moral Reasoning ? How the Mind Interprets Social Exchange Rules and Detects Cheaters , 2007 .

[117]  D. Eddy.  Judgment under uncertainty: Probabilistic reasoning in clinical medicine: Problems and opportunities , 1982 .

[118]  Douglas G Altman,et al.  Meta-analyses involving cross-over trials: methodological issues. , 2002, International journal of epidemiology.

[119]  Ruth Beyth-Marom,et al.  The effects of mental steps and compatibility on Bayesian reasoning , 2014, Judgment and Decision Making.

[120]  B. Woolf.  ON ESTIMATING THE RELATION BETWEEN BLOOD GROUP AND DISEASE , 1955, Annals of human genetics.

[121]  Gary L. Brase.  Omissions, conflations, and false dichotomies: Conceptual and empirical problems with the Barbey & Sloman account. , 2007 .

[122]  R. Hertwig,et al.  Decisions from Experience and the Effect of Rare Events in Risky Choice , 2004, Psychological science.

[123]  Gary L. Brase.  How different types of participant payments alter task performance , 2009, Judgment and Decision Making.

[124]  J. Tenenbaum,et al.  Special issue on “Probabilistic models of cognition , 2022 .

[125]  Miroslav Sirota,et al.  Beyond getting the numbers right: what does it mean to be a “successful” Bayesian reasoner? , 2015, Front. Psychol..

[126]  G. Bodanza,et al.  Sets or frequencies? How to help people solve conditional probability problems , 2011 .

[127]  Pierre Dragicevic,et al.  Assessing the Effect of Visualizations on Bayesian Reasoning through Crowdsourcing , 2012, IEEE Transactions on Visualization and Computer Graphics.

[128]  Maya Bar-Hillel,et al.  Representativeness and fallacies of probability judgment , 1984 .

[129]  Gabriella Pravettoni,et al.  Facilitating Effect of Natural Frequencies: Size Does Not Matter , 2009, Perceptual and motor skills.

[130]  Gerd Gigerenzer,et al.  Communicating Statistical Information , 2000, Science.

[131]  Michelle Phillipov.  COMMUNICATING HEALTH RISKS VIA THE MEDIA , 2012 .

[132]  A. Sutton,et al.  Comparison of two methods to detect publication bias in meta-analysis. , 2006, JAMA.

[133]  V. Girotto,et al.  Solving probabilistic and statistical problems: a matter of information structure and question form , 2001, Cognition.

[134]  Miroslav Sirota,et al.  Now you Bayes, now you don’t: effects of set-problem and frequency-format mental representations on statistical reasoning , 2015, Psychonomic bulletin & review.

[135]  Shira Elqayam,et al.  Dual-processing explains base-rate neglect, but which dual-process theory and how? , 2007 .

[136]  Michael Siegrist,et al.  Natural frequencies and Bayesian reasoning: the impact of formal education and problem context , 2011 .

[137]  Jingjing Liu,et al.  Numeracy, frequency, and Bayesian reasoning , 2009, Judgment and Decision Making.

[138]  S. Sloman,et al.  Base-rate respect: From ecological rationality to dual processes. , 2007, The Behavioral and brain sciences.

[139]  Gary L. Brase.  Pictorial representations in statistical reasoning , 2009 .

[140]  Yoav Ben-Shlomo,et al.  Presentation of Diagnostic Information to Doctors May Change Their Interpretation and Clinical Management: A Web-Based Randomised Controlled Trial , 2015, PloS one.

[141]  Vittorio Girotto,et al.  Chances and frequencies in probabilistic reasoning: rejoinder to Hoffrage, Gigerenzer, Krauss, and Martignon , 2002, Cognition.

[142]  Richard Samuels,et al.  Varieties of dual-process theory for probabilistic reasoning , 2007 .

[143]  K. Fiedler,et al.  A sampling approach to biases in conditional probability judgments: beyond base rate neglect and statistical format. , 2000, Journal of experimental psychology. General.

[144]  Miroslav Sirota,et al.  The effect of iconicity of visual displays on statistical reasoning: evidence in favor of the null hypothesis , 2014, Psychonomic bulletin & review.

[145]  Gerd Gigerenzer,et al.  Natural Frequencies Help Older Adults and People with Low Numeracy to Evaluate Medical Screening Tests , 2009, Medical decision making : an international journal of the Society for Medical Decision Making.

[146]  Ulrich Hoffrage,et al.  Teaching Bayesian reasoning: an evaluation of a classroom tutorial for medical students , 2002, Medical teacher.

[147]  Ulrich Hoffrage,et al.  Visual representation of statistical information improves diagnostic inferences in doctors and their patients. , 2013, Social science & medicine.

[148]  Gary L. Brase,et al.  Frequency interpretation of ambiguous statistical information facilitates Bayesian reasoning , 2008, Psychonomic bulletin & review.

[149]  S. Thompson,et al.  Quantifying heterogeneity in a meta‐analysis , 2002, Statistics in medicine.

[150]  J. Schreiber.  Foundations Of Statistics , 2016 .

[151]  Gary L. Brase,et al.  When and for whom do frequencies facilitate performance? On the role of numerical literacy , 2012, Quarterly journal of experimental psychology.