Measuring Teaching Quality in Higher Education: Assessing the Problem of Selection Bias in Course Evaluations

Student evaluations of teaching are widely used to measure teaching quality and compare it across different courses, teachers, departments and institutions: as such, they are of increasing importance for teacher promotion decisions as well as student course selection. However, the response on course evaluations is rarely perfect, rendering such uses unwarranted if students who participate in the evaluation are not randomly selected: this paper is the first to investigate this issue. We quantify the direction and size of selection on both observable and unobservable characteristics for a large European university where course evaluation response rates differ across courses. Our results suggest course evaluations are upward biased, and that this bias mostly derives from selection on characteristics unlikely to be observed by the typical university administrator. Correcting for selection bias has sizable effects on both scores in any given course and the evaluation-based ranking of different courses.

[1]  Dennis E. Clayson,et al.  Student Evaluations of Teaching: Are They Related to What Students Learn? , 2009 .

[2]  Jeffrey M. Wooldridge,et al.  Solutions Manual and Supplementary Materials for Econometric Analysis of Cross Section and Panel Data , 2003 .

[3]  Trey Miller,et al.  Measuring value-added in higher education: Possibilities and limitations in the use of administrative data , 2014 .

[4]  Myungsup Kim,et al.  What Determines Student Evaluation Scores? A Random Effects Analysis of Undergraduate Economics Classes , 2009 .

[5]  Bruce A. Weinberg,et al.  Evaluating Teaching in Higher Education , 2009 .

[6]  S. Rabe-Hesketh,et al.  Maximum Likelihood Estimation of Endogenous Switching and Sample Selection Models for Binary, Ordinal, and Count Variables , 2006 .

[7]  Philip Oreopoulos,et al.  Professor Qualities and Student Achievement , 2006, The Review of Economics and Statistics.

[8]  Giuseppe De Luca,et al.  Estimation of Ordered Response Models with Sample Selection , 2010 .

[9]  L. Langbein.  Management by Results: Student Evaluation of Faculty Teaching and the Mis-Measurement of Performance. , 2008 .

[10]  Russel L. Thompson,et al.  A Meta-Analysis of Response Rates in Web- or Internet-Based Surveys , 2000 .

[11]  Anthony C. Krautmann,et al.  Grades and student evaluations of teachers , 1999 .

[12]  William E. Becker,et al.  How Departments of Economics Evaluate Teaching , 1999 .

[13]  Anne E. Sartori.  An Estimator for Some Binary-Outcome Selection Models Without Exclusion Restrictions , 2003, Political Analysis.

[14]  D. D. Morley,et al.  Claims about the reliability of student evaluations of instruction: The ecological fallacy rides again , 2012 .

[15]  M. Pellizzari,et al.  Evaluating Students' Evaluations of Professors , 2011, SSRN Electronic Journal.

[16]  A. Greenwald,et al.  Grading leniency is a removable contaminant of student ratings. , 1997, The American psychologist.

[17]  Michael F. Lovenheim,et al.  Does the Market Value Value-Added? Evidence from Housing Prices after a Public Release of School and Teacher Value-Added , 2013, SSRN Electronic Journal.

[18]  Raj Chetty,et al.  Measuring the Impacts of Teachers I: Evaluating Bias in Teacher Value-Added Estimates , 2014 .

[19]  Michael A. McPherson.  Determinants of How Students Evaluate Teachers , 2006 .

[20]  W. V. D. Ven,et al.  The demand for deductibles in private health insurance: A probit model with sample selection , 1981 .

[21]  J. Heckman.  Dummy Endogenous Variables in a Simultaneous Equation System , 1977 .

[22]  Andrew M. Ewing.  Estimating the impact of relative expected grade on student evaluations of teachers , 2012 .

[23]  Harinder Singh,et al.  Do Higher Grades Lead to Favorable Student Evaluations? , 2005 .