What Makes Everyday Scientific Reasoning So Challenging

Abstract Informed citizens are expected to use science-based evidence to make decisions about health, behavior and public policy. To do so, they must judge whether the evidence is consistent with the claims presented (theory-evidence coordination). Unfortunately, most individuals make numerous errors in theory-evidence coordination. In this chapter, we provide an overview of research on science evidence evaluation, drawing from research in cognitive and developmental psychology, science and statistics education, decision sciences, political science and science communication. Given the breadth of this research area, we highlight some influential studies and reviews across these different topics. This body of research provides several clues about: (1) why science evidence evaluation is challenging, (2) the influence of the content and context of the evidence and (3) how the characteristics of the individual examining the evidence impact the quality of the evaluations. Finally, we suggest some possible directions for empirical research on improving evidence evaluation and point to the responsibility of scientists, especially social and behavioral scientists, in communicating their findings to the public. Overall, our goal is to give readers an interdisciplinary view of science evidence evaluation research and to integrate research from different scientific communities that address similar questions.

[1]  Eric R. Stone,et al.  Effects of numerical and graphical displays on professed risk-taking behavior. , 1997 .

[2]  J. Concato,et al.  Randomized, controlled trials, observational studies, and the hierarchy of research designs. , 2000, The New England journal of medicine.

[3]  Richard Lehrer,et al.  Modeling Natural Variation Through Distribution , 2004 .

[4]  K. Stanovich,et al.  The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks , 2011, Memory & cognition.

[5]  Lin Ding,et al.  Learning and Scientific Reasoning , 2009, Science.

[6]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[7]  Alan D. Castel,et al.  Seeing is believing: The effect of brain images on judgments of scientific reasoning , 2008, Cognition.

[8]  P. Pintrich,et al.  The Development of Epistemological Theories: Beliefs About Knowledge and Knowing and Their Relation to Learning , 1997 .

[9]  Lewis Bott,et al.  The association between exaggeration in health-related science news and academic press releases: a replication study , 2019, Wellcome open research.

[10]  Harriet Shaklee,et al.  Cause and covariate: Development of two related concepts , 1988 .

[11]  E. Soloway,et al.  Enacting Project-Based Science , 1997, The Elementary School Journal.

[12]  Michael Pressley,et al.  Skilled comprehension and its development through instruction. , 1997 .

[13]  Philip H. Winne,et al.  Effects of teaching statistical laws on reasoning about everyday problems. , 1995 .

[14]  Petroc Sumner,et al.  The association between exaggeration in health related science news and academic press releases: retrospective observational study , 2014, BMJ : British Medical Journal.

[15]  J. Baron Thinking and deciding, 4th ed. , 2008 .

[16]  Yaacov Schul,et al.  The value of distrust , 2008 .

[17]  K. Stanovich,et al.  Reasoning independently of prior belief and individual differences in actively open-minded thinking. , 1997 .

[18]  Jessica Daecher Research Methods Designing And Conducting Research With A Real World Focus , 2016 .

[19]  Ian J. Deary,et al.  What Intelligence Tests Miss: The Psychology of Rational Thought , 2010 .

[20]  Priti Shah,et al.  Explaining the alluring influence of neuroscience information on scientific reasoning. , 2014, Journal of experimental psychology. Learning, memory, and cognition.

[21]  K. Stanovich,et al.  The domain specificity and generality of belief bias: Searching for a generalizable critical thinking skill. , 1999 .

[22]  M. Prasad,et al.  "There Must Be a Reason": Osama, Saddam, and Inferred Justification , 2009 .

[23]  Alex Broadbent,et al.  Causality and causal inference in epidemiology: the need for a pluralistic approach , 2016, International journal of epidemiology.

[24]  Deena Skolnick Weisberg,et al.  The Seductive Allure of Neuroscience Explanations , 2008, Journal of Cognitive Neuroscience.

[25]  K. Kitchener,et al.  Developing Reflective Judgment: Understanding and Promoting Intellectual Growth and Critical Thinking in Adolescents and Adults. Jossey-Bass Higher and Adult Education Series and Jossey-Bass Social and Behavioral Science Series. , 2009 .

[26]  M. Slater,et al.  How Message Evaluation and Source Attributes May Influence Credibility Assessment and Belief Change , 1996 .

[27]  Z. Kunda,et al.  The case for motivated reasoning. , 1990, Psychological bulletin.

[28]  M. Schaller,et al.  "Intuitive Analysis of Covariance" and Group Stereotype Formation , 1992 .

[29]  Anton E. Lawson,et al.  The development and validation of a classroom test of formal reasoning , 1978 .

[30]  J. Osborne,et al.  Bridging science education and science communication research , 2015 .

[31]  R. Mccall,et al.  The Genetic and Environmental Origins of Learning Abilities and Disabilities in the Early School , 2007, Monographs of the Society for Research in Child Development.

[32]  Julie Hatfield,et al.  Avoiding confusion surrounding the phrase "correlation does not imply causation." [References] , 2006 .

[33]  D. Perkins,et al.  Beyond Abilities: A Dispositional Theory of Thinking. , 1993 .

[34]  Dominique Brossard,et al.  Deference to Scientific Authority Among a Low Information Public: Understanding U.S. Opinion on Agricultural Biotechnology , 2006 .

[35]  Edward T. Cokely,et al.  Measuring Risk Literacy: The Berlin Numeracy Test , 2012, Judgment and Decision Making.

[36]  Karen L. Siedlecki,et al.  Individual differences in Need for Cognition influence the evaluation of circular scientific explanations , 2016 .

[37]  J. Osborne,et al.  Teaching and Learning Science as Argument , 2010 .

[38]  D. Garside Philosophy for children , 2014 .

[39]  J. Baron,et al.  Confusion of Relative and Absolute Risk in Valuation , 1997 .

[40]  B. Koslowski Theory and Evidence: The Development of Scientific Reasoning , 1996 .

[41]  Ngss Lead States Next generation science standards : for states, by states , 2013 .

[42]  D. Klahr,et al.  All other things being equal: acquisition and transfer of the control of variables strategy. , 1999, Child development.

[43]  Edward T. Cokely,et al.  Cognitive abilities and superior decision making under risk: A protocol analysis and process model evaluation , 2009, Judgment and Decision Making.

[44]  Norbert Schwarz,et al.  Making the Truth Stick & the Myths Fade: Lessons from Cognitive Psychology , 2016, Behavioral Science & Policy.

[45]  Diego Fernandez-Duque,et al.  Superfluous Neuroscience Information Makes Explanations of Psychological Phenomena More Appealing , 2015, Journal of Cognitive Neuroscience.

[46]  Wim De Neys,et al.  Conflict monitoring in dual process theories of thinking , 2008, Cognition.

[47]  G. Munro,et al.  The Scientific Impotence Excuse:Discounting Belief-Threatening Scientific Abstracts , 2010 .

[48]  Aner Tal,et al.  Blinded with science: Trivial graphs and formulas increase ad persuasiveness and belief in product efficacy , 2016, Public understanding of science.

[49]  Eugene Borgida,et al.  The Differential Impact of Abstract vs. Concrete Information on Decisions , 1977 .

[50]  L. Schauble,et al.  Scientific Thinking and Science Literacy , 2007 .

[51]  Kimmo Eriksson The nonsense math effect , 2012, Judgment and Decision Making.

[52]  J. Trefil Science Education for Everyone: Why and What?. , 2008 .

[53]  J. Ioannidis Why Most Discovered True Associations Are Inflated , 2008, Epidemiology.

[54]  H. Chua,et al.  Risk avoidance: Graphs versus numbers , 2006, Memory & cognition.

[55]  B. Rimer,et al.  General Performance on a Numeracy Scale among Highly Educated Samples , 2001, Medical decision making : an international journal of the Society for Medical Decision Making.

[56]  M. Lipman PHILOSOPHY FOR CHILDREN , 1991 .

[57]  Shoshana Shiloh,et al.  Individual differences in rational and intuitive thinking styles as predictors of heuristic responses and framing effects , 2002 .

[58]  Gregory N. Mandel,et al.  The Polarizing Impact of Science Literacy and Numeracy on Perceived Climate Change Risks , 2012 .

[59]  P. Klaczynski,et al.  Goal-oriented critical reasoning and individual differences in critical reasoning biases. , 1997 .

[60]  L. Marschall The Demon-Haunted World , 1996 .

[61]  K. Stanovich,et al.  Myside Bias, Rational Thinking, and Intelligence , 2013 .

[62]  T. Lombrozo The structure and function of explanations , 2006, Trends in Cognitive Sciences.

[63]  A. Newell You can't play 20 questions with nature and win : projective comments on the papers of this symposium , 1973 .

[64]  A. Anastasi Individual differences. , 2020, Annual review of psychology.

[65]  S. Sloman When explanations compete: the role of explanatory coherence on judgements of likelihood , 1994, Cognition.

[66]  Kassandra I. Alcaraz,et al.  Comparing narrative and informational videos to increase mammography in low-income African American women. , 2010, Patient education and counseling.

[67]  Joan Garfield ASSESSING STATISTICAL REASONING , 2003, STATISTICS EDUCATION RESEARCH JOURNAL.

[68]  Yair Neuman,et al.  Pupils' evaluation and generation of evidence and explanation in argumentation. , 2005, The British journal of educational psychology.

[69]  R. Yager Science/Technology/Society as Reform in Science Education. , 1996 .

[70]  D. Medin,et al.  The role of theories in conceptual coherence. , 1985, Psychological review.

[71]  M. Wake,et al.  Academic Outcomes 2 Years After Working Memory Training for Children With Low Working Memory: A Randomized Clinical Trial. , 2016, JAMA pediatrics.

[72]  Winston R. Sieck,et al.  Foreground:background salience: Explaining the effects of graphical displays on risk avoidance , 2003 .

[73]  D. Kuhn,et al.  Is Developing Scientific Thinking All About Learning to Control Variables? , 2005, Psychological science.

[74]  Michael Siegrist,et al.  Knowledge as a driver of public perceptions about climate change reassessed , 2016 .

[75]  Jessica Thompson,et al.  Beyond the scientific method: Model‐based inquiry as a new paradigm of preference for school science investigations , 2008 .

[76]  D. R. Lehman,et al.  Teaching reasoning. , 1987, Science.

[77]  D. Krantz,et al.  The effects of statistical training on thinking about everyday problems , 1986, Cognitive Psychology.

[78]  P. Ubel,et al.  Reducing the Influence of Anecdotal Reasoning on People’s Health Care Decisions: Is a Picture Worth a Thousand Statistics? , 2005, Medical decision making : an international journal of the Society for Medical Decision Making.

[79]  L. Miller,et al.  Research Literacy: A Primer for Understanding and Using Research , 2016 .

[80]  Yaacov Trope,et al.  Problem Solving in Judgment Under Uncertainty , 1987 .

[81]  Hollyn M. Johnson,et al.  Sources of the continued influence effect: When misinformation in memory affects later inferences. , 1994 .

[82]  Derek Powell,et al.  Countering antivaccination attitudes , 2015, Proceedings of the National Academy of Sciences.

[83]  Thomas I. Ellis Teaching Thinking Skills. , 1988 .

[84]  C. Schunn,et al.  Unpacking the Relationship Between Science Education and Applied Scientific Literacy , 2016 .

[85]  E. Peters Beyond Comprehension , 2012 .

[86]  V. Reyna,et al.  How numeracy influences risk comprehension and medical decision making. , 2009, Psychological bulletin.

[87]  B. Koslowski,et al.  Information Becomes Evidence when an Explanation Can Incorporate It into a Causal Framework. , 2008 .

[88]  Susan R. Goldman,et al.  The Public's Bounded Understanding of Science , 2014 .

[89]  D. Kuhn How do People Know? , 2001, Psychological science.

[90]  Vetta L. Sanders Thompson,et al.  Making decisions in a complex information environment: evidential preference and information we trust , 2013, BMC Medical Informatics Decis. Mak..

[91]  Nathan R. Kuncel,et al.  Does College Teach Critical Thinking? A Meta-Analysis , 2016 .

[92]  Ian J. Deary,et al.  Teaching intelligence , 2013 .

[93]  S. Heine,et al.  Genetic essentialism: on the deceptive determinism of DNA. , 2011, Psychological bulletin.

[94]  S. Frederick Journal of Economic Perspectives—Volume 19, Number 4—Fall 2005—Pages 25–42 Cognitive Reflection and Decision Making , 2022 .

[95]  D. Rubin,et al.  Contrasts and Effect Sizes in Behavioral Research: A Correlational Approach , 1999 .

[96]  John R. Anderson,et al.  The generality/specificity of expertise in scientific reasoning , 1999, Cogn. Sci..

[97]  Jonathan A. Fugelsang,et al.  Analytic cognitive style predicts religious and paranormal belief , 2012, Cognition.

[98]  B. Nyhan,et al.  When Corrections Fail: The Persistence of Political Misperceptions , 2010 .

[99]  Rebecca E. Rhodes,et al.  Examining the influence of anecdotal stories and the interplay of individual differences on reasoning , 2016 .

[100]  D. Medin,et al.  The role of covariation versus mechanism information in causal attribution , 1995, Cognition.

[101]  C. Anelli Scientific Literacy: What Is It, Are We Teaching It, and Does It Matter? , 2011 .

[102]  Leonid Rozenblit,et al.  The misunderstood limits of folk science: an illusion of explanatory depth , 2002, Cogn. Sci..

[103]  D. Kuhn,et al.  Social science as a tool in developing scientific thinking skills in underserved, low-achieving urban students. , 2016, Journal of experimental child psychology.

[104]  P. Klaczynski,et al.  A dual-process account of the development of scientific reasoning: The nature and development of metacognitive intercession skills , 2008 .

[105]  Gerd Gigerenzer,et al.  Intuitions About Sample Size: The Empirical Law of Large Numbers , 1997 .

[106]  Ian Skurnik,et al.  Metacognitive Experiences and the Intricacies of Setting People Straight: Implications for Debiasing and Public Information Campaigns , 2007 .

[107]  R. Bromme,et al.  Dealing With Uncertainty: Readers' Memory for and Use of Conflicting Information From Science Texts as Function of Presentation Format and Source Expertise , 2013 .

[108]  Peter H. Ditto,et al.  Motivated Skepticism: Use of Differential Decision Criteria for Preferred and Nonpreferred Conclusions , 1992 .

[109]  Marie S. Burrage "That's an Interesting Finding, but...:" Postsecondary Students' Interpretations of Research Findings. , 2008 .

[110]  Deanna Kuhn,et al.  Coordinating own and other perspectives in argument , 2007 .

[111]  Harrison Si,et al.  Handbook of Research Methods in Social and Personality Psychology: Author Index , 2013 .

[112]  Connie A. Korpan,et al.  University Students' Interpretation of Media Reports of Science and its Relationship to Background Knowledge, Interest, and Reading Difficulty , 2003 .

[113]  G. Domhoff,et al.  Much Ado About Very Little: The Small Effect Sizes When Home and Laboratory Collected Dreams Are Compared , 1999 .

[114]  Vincent Aleven,et al.  Constructing Causal Diagrams to Learn Deliberation , 2009, Int. J. Artif. Intell. Educ..

[115]  Lisa M. Schwartz,et al.  The Role of Numeracy in Understanding the Benefit of Screening Mammography , 1997, Annals of Internal Medicine.

[116]  J. Baron Thinking and Deciding , 2023 .

[117]  W. Ahn,et al.  The Role of Mechanism Beliefs in Causal Reasoning , 2000 .

[118]  Mark Schaller,et al.  In-group favoritism and statistical reasoning in social inference : implications for formation and maintenance of group stereotypes , 1992 .

[119]  David Evans,et al.  Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. , 2003, Journal of clinical nursing.

[120]  Stephen P. Norris,et al.  Interpreting pragmatic meaning when reading popular reports of science , 1994 .

[121]  Paul Slovic,et al.  Motivated numeracy and enlightened self-government , 2017, Behavioural Public Policy.

[122]  Jon D. Miller,et al.  Public Acceptance of Evolution , 2006, Science.

[123]  Frank Renkewitz,et al.  The Influence of Narrative v. Statistical Information on Perceiving Vaccination Risks , 2011, Medical decision making : an international journal of the Society for Medical Decision Making.

[124]  Emily J. Hopkins,et al.  The seductive allure is a reductive allure: People prefer scientific explanations that contain logically irrelevant reductive information , 2016, Cognition.

[125]  M. Anne Britt,et al.  The seduction of easiness: How science depictions influence laypeople’s reliance on their own evaluation of scientific information , 2012 .

[126]  Jonathan Evans,et al.  Rapid responding increases belief bias: Evidence for the dual-process theory of reasoning , 2005 .

[127]  D. Halpern Teaching critical thinking for transfer across domains. Dispositions, skills, structure training, and metacognitive monitoring. , 1998, The American psychologist.

[128]  R. Nickerson,et al.  Teaching Thinking Skills , 1986 .

[129]  George W Rebok,et al.  Effects of cognitive training interventions with older adults: a randomized controlled trial. , 2002, JAMA.

[130]  Colleen M. Seifert,et al.  The continued influence of misinformation in memory: What makes a correction effective? , 2002 .

[131]  S. Epstein,et al.  Individual differences in intuitive-experiential and analytical-rational thinking styles. , 1996, Journal of personality and social psychology.

[132]  L. Mason Role of anomalous data and epistemological beliefs in middle school students’ theory change about two controversial topics , 2000 .

[133]  Richard Lehrer,et al.  Supporting the Development of Conceptions of Statistics by Engaging Students in Measuring and Modeling Variability , 2007, Int. J. Comput. Math. Learn..

[134]  W. Sandoval Understanding Students' Practical Epistemologies and Their Influence on Learning Through Inquiry , 2005 .

[135]  P. Cheng,et al.  Distinguishing Genuine from Spurious Causes: A Coherence Hypothesis , 2000, Cognitive Psychology.

[136]  Richard M Shiffrin,et al.  Drawing causal inference from Big Data , 2016, Proceedings of the National Academy of Sciences.

[137]  D. Beck Science Perspectives on Psychological the Appeal of the Brain in the Popular Press on Behalf Of: Association for Psychological Science , 2022 .

[138]  Uwe Oestermeier,et al.  Verbal and visual causal arguments , 2000, Cognition.

[139]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[140]  Keith E. Stanovich,et al.  A framework for critical thinking, rational thinking, and intelligence. , 2010 .

[141]  Hans Knutsson,et al.  Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates , 2016, Proceedings of the National Academy of Sciences.

[142]  G. Sinatra,et al.  Addressing Challenges to Public Understanding of Science: Epistemic Cognition, Motivated Reasoning, and Conceptual Change , 2014 .

[143]  Rebecca E. Rhodes The Influence of Reductionist Information on Perceptions of Scientific Validity. , 2015 .

[144]  Anders Isnes,et al.  Science students' critical examination of scientific information related to socioscientific issues , 2006 .

[145]  Emily E. Dunlap,et al.  Using Valid and Invalid Experimental Designs to Teach the Control of Variables Strategy in Higher and Lower Achieving Classrooms. , 2014 .

[146]  David Klahr,et al.  Dual Space Search During Scientific Reasoning , 1988, Cogn. Sci..

[147]  Yaacov Schul,et al.  When warning succeeds: The effect of warning on success in ignoring invalid information. , 1993 .

[148]  Michael B. Salwen,et al.  Speaking into the Air: A History of the Idea of Communication , 2000 .

[149]  Richard Lehrer,et al.  Exploring Children's Data Modeling , 1996 .

[150]  P. Ubel,et al.  Measuring Numeracy without a Math Test: Development of the Subjective Numeracy Scale , 2007, Medical decision making : an international journal of the Society for Medical Decision Making.

[151]  H. Pashler,et al.  Is the Replicability Crisis Overblown? Three Arguments Examined , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[152]  J. Cacioppo,et al.  The need for cognition. , 1982 .

[153]  Alan H. Schoenfeld,et al.  Explicit Heuristic Training as a Variable in Problem-Solving Performance**Chapter 6 is an expanded and revised version of Schoenfeld (1979b). Permission from the Journal for Research in Mathematics Education to reproduce parts of the article is gratefully acknowledged. , 1979 .

[154]  Barbara Koslowski,et al.  When covariation is not enough: The role of causal mechanism, sampling method, and sample size in causal reasoning. , 1989 .

[155]  Cayce J. Hook,et al.  The Seductive Allure of “Seductive Allure” , 2013, Perspectives on psychological science : a journal of the Association for Psychological Science.

[156]  Milena K. Nigam,et al.  The Equivalence of Learning Paths in Early Science Instruction: Effects of Direct Instruction and Discovery Learning , 2022 .

[157]  N. McGlynn Thinking fast and slow. , 2014, Australian veterinary journal.

[158]  Leif D. Nelson,et al.  False-Positive Psychology , 2011, Psychological science.

[159]  D. Kuhn Strategies of Knowledge Acquisition , 1995 .

[160]  S. Croker,et al.  Teaching the control-of-variables strategy: A meta-analysis , 2016 .

[161]  Jonathan Evans In two minds: dual-process accounts of reasoning , 2003, Trends in Cognitive Sciences.

[162]  Jon D. Miller,et al.  Science communication. Public acceptance of evolution. , 2006, Science.

[163]  R. Nisbett,et al.  Immediate and delayed transfer of training effects in statistical reasoning. , 1991, Journal of experimental psychology. General.

[164]  Paul R. Pintrich,et al.  Development and Validation of the Epistemic Belief Inventory (EBI) , 2001 .

[165]  P. Klaczynski,et al.  Motivated scientific reasoning biases, epistemological beliefs, and theory polarization: a two-process approach to adolescent cognition. , 2000, Child development.

[166]  Martha J. Farah,et al.  Look Again: Effects of Brain Images and Mind–Brain Dualism on Lay Evaluations of Research , 2013, Journal of Cognitive Neuroscience.

[167]  R. Nickerson Confirmation Bias: A Ubiquitous Phenomenon in Many Guises , 1998 .

[168]  R. H. Walters The Growth of Logical Thinking from Childhood to Adolescence , 1960 .

[169]  M. Kendall,et al.  The Logic of Scientific Discovery. , 1959 .

[170]  J Baron,et al.  The Inclusion of Patient Testimonials in Decision Aids , 2001, Medical decision making : an international journal of the Society for Medical Decision Making.

[171]  L. Ross,et al.  Biased Assimilation and Attitude Polarization: The Effects of Prior Theories on Subsequently Considered Evidence , 1979 .

[172]  Z. Kunda,et al.  Ducking the collection of costly evidence: Motivated use of statistical heuristics , 1991 .

[173]  A. Tversky,et al.  Subjective Probability: A Judgment of Representativeness , 1972 .

[174]  B. Reiser,et al.  Developing a learning progression for scientific modeling: Making scientific modeling accessible and meaningful for learners , 2009 .

[175]  Gary James Jason,et al.  The Logic of Scientific Discovery , 1988 .