Students’ misconceptions of statistical inference: A review of the empirical evidence from research on statistics education

Abstract A solid understanding of inferential statistics is of major importance for designing and interpreting empirical results in any scientific discipline. However, students are prone to many misconceptions regarding this topic. This article structurally summarizes and describes these misconceptions by presenting a systematic review of publications that provide empirical evidence of them. This group of publications was found to be dispersed over a wide range of specialized journals and proceedings, and the methodology used in the empirical studies was very diverse. Three research needs rise from this review: (1) further empirical studies that identify the sources and possible solutions for misconceptions in order to complement the abundant theoretical and statistical discussion about them; (2) new insights into effective research designs and methodologies to perform this type of research; and (3) structured and systematic summaries of findings like the one presented here, concerning misconceptions in other areas of statistics, that might be of interest both for educational researchers and teachers of statistics.

[1]  Larry G. Daniel,et al.  Statistical Significance Testing: A Historical Overview of Misuse and Misinterpretation with Implications for the Editorial Policies of Educational Journals , 1998 .

[2]  E. Glasersfeld.  Radical constructivism in mathematics education , 2002 .

[3]  D. Krantz.  The Null Hypothesis Testing Controversy in Psychology , 1999 .

[4]  A. Vallecillos.  Understanding of the Logic of Hypothesis Testing Amongst University Students , 2000 .

[5]  D. Schifter,et al.  A research companion to Principles and standards for school mathematics , 2003 .

[6]  Some Misconceptions about the Normal Distribution , 2003 .

[7]  Carmen Batanero,et al.  Significado e compreensão da distribuição normal em um curso introdutório de análise de dados , 2001 .

[8]  Joan Garfield.  ASSESSING STATISTICAL REASONING , 2003, STATISTICS EDUCATION RESEARCH JOURNAL.

[9]  A. V. Jiménez.  Comprensión de la lógica del contraste de hipótesis en estudiantes universitarios , 1995 .

[10]  Ruma Falk.  Misconceptions of statistical significance. , 1986 .

[11]  Heiko Haller,et al.  Misinterpretations of significance: A problem students share with their teachers? , 2002 .

[12]  Gerd Gigerenzer,et al.  The superego, the ego, and the id in statistical reasoning , 1993 .

[13]  G. A. Miller,et al.  Book Review Nisbett, R. , & Ross, L.Human inference: Strategies and shortcomings of social judgment.Englewood Cliffs, N.J.: Prentice-Hall, 1980. , 1982 .

[14]  L. Harlow,et al.  What if there were no significance tests , 1997 .

[15]  Jane Watson,et al.  Development of understanding of sampling for statistical literacy , 2000 .

[16]  M. Ware,et al.  Developing Selection Skills in Introductory Statistics , 1991 .

[17]  C. B. Bernabéu,et al.  Conceptos activados en el contraste de hipótesis estadisticas y su comprensión por estudiantes universitarios , 1997 .

[18]  E. Ziegel.  Introduction to the Practice of Statistics (2nd ed.) , 1994 .

[19]  Roger E. Kirk,et al.  Promoting Good Statistical Practices: Some Suggestions , 2001 .

[20]  Uri Wilensky,et al.  What is Normal Anyway? Therapy for Epistemological Anxiety , 1997 .

[21]  A. Tversky,et al.  Subjective Probability: A Judgment of Representativeness , 1972 .

[22]  Carmen Batanero.  Statistics Education as a Field for Research and Practice , 2005 .

[23]  N. Leech,et al.  Problems With Null Hypothesis Significance Testing (NHST): What Do the Textbooks Say? , 2002 .

[24]  Gerd Gigerenzer,et al.  Was Bernoulli wrong? On intuitions about sample size , 2000 .

[25]  Aarnout Brombacher,et al.  Probability... , 2009, Qual. Reliab. Eng. Int..

[26]  Behavioral Statistics Textbooks: Source of Myths and Misconceptions? , 1985 .

[27]  Jessica S. Ancker,et al.  The Language of Conditional Probability , 2006 .

[28]  Liliana Tauber,et al.  Students’ Reasoning about the Normal Distribution , 2004 .

[29]  UNDERSTANDING REPLICATION: CONFIDENCE INTERVALS, p VALUES, AND WHAT'S LIKELY TO HAPPEN NEXT TIME , 2006 .

[30]  Jeremy Kilpatrick,et al.  International handbook of mathematics education , 1997 .

[31]  Douglas A. Grouws,et al.  Handbook of research on mathematics teaching and learning , 1992 .

[32]  James E. McLean,et al.  The Role of Statistical Significance Testing In Educational Research , 1998 .

[33]  C. Konold,et al.  Informal Conceptions of Probability , 1989 .

[34]  H. J. Arnold.  Introduction to the Practice of Statistics , 1990 .

[35]  G. Cumming,et al.  Editors Can Lead Researchers to Confidence Intervals, but Can't Make Them Think , 2004, Psychological science.

[36]  Angustias Vallecillos,et al.  SOME EMPIRICAL EVIDENCES ON LEARNING DIFFICULTIES ABOUT TESTING HYPOTHESES , 1999 .

[37]  Carmen Batanero,et al.  Errors and difficulties in understanding elementary statistical concepts , 1994 .

[38]  Barbara J. Guzzetti,et al.  Promoting conceptual change in science: A comparative meta-analysis of instructional interventions from reading education and science education , 1993 .

[39]  Jacques Poitevineau,et al.  Even statisticians are not immune to misinterpretations of Null Hypothesis Significance Tests , 2003 .

[40]  G. Gigerenzer.  On Narrow Norms and Vague Heuristics: A Reply to Kahneman and Tversky (1996) , 1996 .

[41]  L. Ross,et al.  Human Inference: Strategies and Shortcomings of Social Judgment. , 1981 .

[42]  R. Falk,et al.  Significance Tests Die Hard , 1995 .

[43]  Christopher L. Aberson,et al.  Evaluation of an Interactive Tutorial for Teaching Hypothesis Testing Concepts , 2003 .

[44]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[45]  J. Shaughnessy,et al.  Monty's Dilemma: Should You Stick or Switch?. , 1991 .

[46]  Clifford Konold,et al.  Understanding Students’ Beliefs About Probability , 1991 .

[47]  Bruce Thompson,et al.  A National Survey of AERA Members' Perceptions of Statistical Significance Tests and Other Statistical Issues. , 2000 .

[48]  Peter Sedlmeier,et al.  The distribution matters: two types of sample-size tasks , 1998 .

[49]  Nils J. Nilsson,et al.  Probabilistic Logic * , 2022 .

[50]  Gideon Keren,et al.  A Handbook for data analysis in the behavioral sciences : methodological issues , 1993 .

[51]  Barbara J. Dougherty,et al.  Mathematics Education Across Cultures: Proceedings of the 42nd Meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education , 2020 .

[52]  Beth Chance,et al.  Reasoning about Sampling Distribitions , 2004 .

[53]  J. Healey.  Statistics: A Tool for Social Research , 1995 .

[54]  J. Shaughnessy.  Research in probability and statistics: Reflections and directions. , 1992 .

[55]  D. Cox,et al.  Statistical significance tests. , 1982, British journal of clinical pharmacology.

[56]  Joan Garfield,et al.  The challenge of developing statistical literacy, reasoning and thinking , 2004 .

[57]  Geoff Cumming,et al.  Confidence intervals and replication: where will the next mean fall? , 2006, Psychological methods.

[58]  Steve Cohen,et al.  Identifying Impediments to Learning Probability and Statistics From an Assessment of Instructional Software , 1996 .

[59]  S. Finch.  Explaining the Law of Large Numbers , 2003 .

[60]  Maya Bar-Hillel,et al.  The role of sample size in sample evaluation , 1979 .

[61]  G. Cumming,et al.  Researchers misunderstand confidence intervals and standard error bars. , 2005, Psychological methods.

[62]  Jacob Cohen,et al.  THINGS I HAVE LEARNED (SO FAR) , 1990 .

[63]  Fiona Fidler,et al.  SHOULD PSYCHOLOGY ABANDON p VALUES AND TEACH CIs INSTEAD? EVIDENCE-BASED REFORMS IN STATISICS EDUCATION , 2006 .

[64]  Alexander Pollatsek,et al.  Understanding the effects of sample size on the variability of the mean , 1990 .

[65]  Yan Liu,et al.  Exploring Students' Conceptions of the Standard Deviation , 2005 .

[66]  HOW SIGNIFICANCE TESTS SHOULD BE PRESENTED TO AVOID THE TYPICAL MISINTERPRETATIONS , 2002 .

[67]  Kay Lipson,et al.  THE ROLE OF COMPUTER BASED TECHNOLOGY IN DEVELOPING UNDERSTANDING OF THE CONCEPT OF SAMPLING DISTRIBUTION , 2002 .

[68]  A. Tversky,et al.  BELIEF IN THE LAW OF SMALL NUMBERS , 1971, Pediatrics.

[69]  Amanda Williams.  Students''understanding of the significance level concept , 1998 .

[70]  G. Cumming,et al.  Replication and Researchers' Understanding of Confidence Intervals and Standard Error Bars. , 2004 .

[71]  Carmen Batanero,et al.  Controversies Around the Role of Statistical Tests in Experimental Research , 2000 .

[72]  Geoff Cumming,et al.  Assessing conceptual change in learning statistics , 1998 .

[73]  J. Roschelle,et al.  Misconceptions Reconceived: A Constructivist Analysis of Knowledge in Transition , 1994 .