Sniffing out the Secret Poison: Selection Bias in Educational Research.

Selection bias is a persistent, and often hidden, problem in educational research. It is the primary obstacle standing in between increasingly available large education datasets and the ability to make valid causal inferences to inform policymaking, research, and practice (Stuart, 2010). This article provides an accessible discussion on the importance of understanding selection bias in educational research. Although a general explanation on how to remove selection bias is beyond the scope of this article, the reader is guided through an example of this removal process. Specifically, a propensity score analysis is used on a nationally representative dataset to examine whether high school course taking in the algebra-calculus pipeline has a causal effect on placing out of postsecondary remedial mathematics. Several visualizations of the selection bias, and the process of its removal, are provided to give readers a sense of its impact on analyzing observational data.

[1]  J. Goodman.  The Labor of Division: Returns to Compulsory Math Coursework , 2012 .

[2]  Guanglei Hong,et al.  Marginal Mean Weighting Through Stratification: Adjustment for Selection Bias in Multilevel Data , 2010 .

[3]  D. Rubin.  INFERENCE AND MISSING DATA , 1975 .

[4]  Guanglei Hong.  Causal inference for multi -level observational data with application to kindergarten retention. , 2004 .

[5]  Elizabeth A Stuart,et al.  Developing practical recommendations for the use of propensity scores: Discussion of ‘A critical appraisal of propensity score matching in the medical literature between 1996 and 2003’ by Peter Austin, Statistics in Medicine , 2008, Statistics in medicine.

[6]  Robert E. Reys,et al.  Two high school mathematics curricular paths-which one to take? , 2009 .

[7]  Gary King,et al.  Misunderstandings between experimentalists and observationalists about causal inference , 2008 .

[8]  A. Mood,et al.  Equality of Educational Opportunity. , 1967 .

[9]  J. Betts,et al.  The Effect of High School Courses on Earnings , 2004, Review of Economics and Statistics.

[10]  Graham K. Rand,et al.  Quantitative Applications in the Social Sciences , 1983 .

[11]  B. McCall,et al.  Completing Algebra II in High School: Does it Increase College Access and Success? , 2015 .

[12]  Rebecca T Barber.  Characteristics of Students Placed in College Remedial Mathematics : Using the ELS 2002/2006 Data to Understand Remedial Mathematics Placements , 2011 .

[13]  E. Ross,et al.  Report of the Committee of Ten , 1912, American Journal of Sociology.

[14]  J. Graham,et al.  Missing data analysis: making it work in the real world. , 2009, Annual review of psychology.

[15]  The effects of high school math curriculum on college attendance: Evidence from the NLSY97 , 2012 .

[16]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[17]  J. Paul Leigh,et al.  Instrumental variables technique: cigarette price provided better estimate of effects of smoking on SF-12. , 2004, Journal of clinical epidemiology.

[18]  Jeff E. Hoyt,et al.  High School Preparation, Placement Testing, and College Remediation , 2001 .

[19]  Guanglei Hong.  Marginal mean weighting through stratification: a generalized method for evaluating multivalued and multiple treatments with nonexperimental data. , 2012, Psychological methods.

[20]  C. Adelman.  Answers In The Tool Box: Academic Intensity, Attendance Patterns, And Bachelor's Degree Attainment , 1999 .

[21]  Barbara Schneider.  Estimating causal effects : using experimental and observational designs : a think tank white paper , 2007 .

[22]  G. Imbens,et al.  Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score , 2002 .

[23]  Mark C. Long,et al.  Explaining Gaps in Readiness for College-Level Math: The Role of High School Courses , 2009, Education Finance and Policy.

[24]  A. Schoenfeld.  Making Mathematics Work for All Children: Issues of Standards, Testing, and Equity , 2002 .

[25]  Eric Gutstein,et al.  Reading And Writing The World With Mathematics: Toward a Pedagogy for Social Justice , 2005 .

[26]  J. Currie.  When Do We Really Know What We Think We Know ? Determining Causality , 2003 .

[27]  Peter M. Steiner,et al.  Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments , 2008 .

[28]  Elizabeth A Stuart,et al.  Matching methods for causal inference: A review and a look forward. , 2010, Statistical science : a review journal of the Institute of Mathematical Statistics.

[29]  Jeff E. Hoyt.  Level of Math Preparation in High School and Its Impact on Remedial Placement at an Urban State College , 1999 .

[30]  Joseph G. Altonji.  The Effects of High School Curriculum on Education and Labor Market Outcomes , 1992 .

[31]  G. Imbens,et al.  Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score , 2000 .

[32]  Jeffrey Roth,et al.  Effect of High School Course-taking and Grades on Passing a College Placement Test , 1999 .

[33]  Michael Lechner,et al.  Some practical issues in the evaluation of heterogeneous labour market programmes by matching methods , 2002 .

[34]  Clive R. Belfield,et al.  Improving the Targeting of Treatment , 2012 .

[35]  C. Ferguson.  An effect size primer: A guide for clinicians and researchers. , 2009 .

[36]  C. Adelman.  Executive Summary: The Toolbox Revisited--Paths to Degree Completion from High School through College. , 2006 .

[37]  James E. Rosenbaum,et al.  The Unintended Consequences of Stigma-free Remediation. , 2002 .

[38]  Gary King,et al.  Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference , 2007, Political Analysis.