How Well Do Multiple Choice Tests Evaluate Student Understanding in Computer Programming Classes?

Despite the wide diversity of formats with which to construct class examinations, there are many reasons why both university students and instructors prefer multiple-choice tests over other types of exam questions. The purpose of the present study was to examine this multiple-choice/constructed-response debate within the context of teaching computer programming classes. This paper reports the analysis of over 150 test scores of students who were given both multiplechoice and short-answer questions on the same midterm examination. We found that, while student performance on these different types of questions was statistically correlated, the scores on the coding questions explained less than half the variability in the scores on the multiple choice questions. Gender, graduate status, and university major were not significant. This paper also provides some caveats in interpreting our results, suggests some extensions to the present work, and perhaps most importantly in light of the uncovered weak statistical relationship, addresses the question of whether multiple-choice tests are “good enough.”

[1]  William B. Walstad,et al.  Teaching Undergraduate Economics: A Handbook for Instructors , 1997 .

[2]  M. Zeidner.  Essay Versus Multiple-Choice Type Classroom Exams: The Student's Perspective. , 1987 .

[3]  Margaret A. Ray,et al.  Performance and Attitudinal Effects of a Computer Laboratory in the Principles of Economics Course , 1992 .

[4]  Yigal Attali,et al.  Seek Whence , 2002 .

[5]  Bulent Uyar,et al.  Student performance in business and economics statistics: Does exam structure matter? , 2001 .

[6]  Patrick B. O'Neill.  Essay versus Multiple Choice Exams; An Experiment in the Principles of Macroeconomics Course , 2001 .

[7]  B. Bloom.  Taxonomy of educational objectives , 1956 .

[8]  Jane S. Lopus,et al.  A Cost Effectiveness Analysis of Large and Small Classes in the University , 1995 .

[9]  William B. Walstad,et al.  Differential Item Functioning and Male-Female Differences on Multiple-Choice Tests in Economics , 1997 .

[10]  K. Scouller.  The influence of assessment method on students' learning approaches: Multiple choice question examination versus assignment essay , 1998 .

[11]  B. Greene,et al.  Verbal Abilities, Gender, and the Introductory Economics Course: A New Look at an Old Assumption , 1997 .

[12]  William E. Becker,et al.  Achievement Differences on Multiple-Choice and Essay Tests in Economics , 1994 .

[13]  Irvin R. Katz,et al.  Effects of Response Format on Difficulty of SAT-Mathematics Items: It's Not the Strategy , 2000 .

[14]  Richard E. Snow.  Construct validity and constructed-response tests. , 1993 .

[15]  Brent Bridgeman,et al.  A Comparison of Quantitative Questions in Open‐Ended and Multiple‐Choice Formats , 1992 .

[16]  H. Wainer,et al.  COMBINING MULTIPLE-CHOICE AND CONSTRUCTED RESPONSE TEST SCORES: TOWARD A MARXIST THEORY OF TEST CONSTRUCTION , 1992 .

[17]  J. Heckman.  Sample selection bias as a specification error , 1979 .

[18]  S. Douglas,et al.  Estimating Educational Production Functions with Correction for Drops , 1995 .

[19]  Peter E. Kennedy,et al.  Are Multiple-Choice Exams Easier for Economics Students? A Comparison of Multiple-Choice and “Equivalent” Constructed-Response Exam Questions , 2002 .

[20]  John J. Siegfried,et al.  Research on Teaching College Economics: A Survey , 1979 .

[21]  Paul W. Grimes,et al.  The Social Issues Pedagogy vs. The Traditional Principles of Economics: An Empirical Examination , 1995 .

[22]  Samuel Messick.  Trait equivalence as construct validity of score interpretation across multiple methods of measurement. , 1993 .

[23]  Robert C. Sykes,et al.  The Scaling of Mixed-Item Format Tests with the One-Parameter and Two-Parameter Partial Credit Models. , 2000 .

[24]  Brent Bridgeman,et al.  The Relationship of Essay and Multiple-Choice Scores with Grades in College Courses. , 1994 .

[25]  William E. Becker,et al.  Data Loss from Pretest to Posttest as a Sample Selection Problem , 1990 .

[26]  Peter E. Kennedy,et al.  Does Pedagogy Vary with Class Size in Introductory Economics , 1995 .

[27]  B. Kniveton,et al.  A correlational analysis of multiple-choice and essay assessment measures , 1996 .

[28]  R. Adams,et al.  Construction Versus Choice in Cognitive Measurement , 1995 .

[29]  Richard L. Tate,et al.  Performance of a Proposed Method for the Linking of Mixed Format Tests With Constructed Response and Multiple Choice Items , 2000 .

[30]  Walter Kintsch,et al.  Propositional and situational representations of text , 1985 .

[31]  Alex Scott,et al.  The Economics Student Reexamined: Male-Female Differences in Comprehension , 1987 .

[32]  B. Bridgeman,et al.  Success in college for students with discrepancies between performance on multiple-choice and essay tests. , 1996 .

[33]  W. Becker,et al.  The Relationship between Multiple Choice and Essay Response Questions in Assessing Economics Understanding , 1999 .