Approaches to Data Analysis of Multiple-Choice Questions.

This paper introduces five commonly used approaches to analyzing multiple-choice test data. They are classical test theory, factor analysis, cluster analysis, item response theory, and model analysis. Brief descriptions of the goals and algorithms of these approaches are provided, together with examples illustrating their applications in physics education research. We minimize mathematics, instead placing emphasis on data interpretation using these approaches.

[1]  T. Kline.  Psychological Testing: A Practical Approach to Design and Evaluation , 2005 .

[2]  Noah S. Podolefsky,et al.  New Instrument for Measuring Student Beliefs about Physics and Learning Physics: The Colorado Learning Attitudes about Science Survey. , 2006 .

[3]  David P Maloney,et al.  Surveying students’ conceptual knowledge of electricity and magnetism , 2001 .

[4]  Paul Kline,et al.  A Handbook of Test Construction : Introduction to Psychometric Design , 1987 .

[5]  C. Spearman.  General intelligence Objectively Determined and Measured , 1904 .

[6]  Rodney L. Doran,et al.  Basic measurement and evaluation of science instruction , 1980 .

[7]  F. Baker.  The basics of item response theory , 1985 .

[8]  Jae-On Kim,et al.  Factor Analysis: Statistical Methods and Practical Issues , 1978 .

[9]  Albert C. Oosterhof.  Classroom applications of educational measurement , 1994 .

[10]  Deborah J. Harris.  Comparison of 1-, 2-, and 3-Parameter IRT Models , 1989 .

[11]  Robert J. Beichner,et al.  Testing student interpretation of kinematics graphs , 1994 .

[12]  Rasil Warnakulasooriya,et al.  Measuring student learning with item response theory , 2008 .

[13]  R. Darrell Bock,et al.  Estimating item parameters and latent ability when responses are scored in two or more nominal categories , 1972 .

[14]  R. Padraic Springuel,et al.  Applying Clustering to Statistical Analysis of Student Reasoning about Two-Dimensional Kinematics. , 2007 .

[15]  Douglas Huffman,et al.  Interpreting the force concept inventory: A reply to Hestenes and Halloun , 1995 .

[16]  Xitao Fan.  Item Response Theory and Classical Test Theory: An Empirical Comparison of their Item/Person Statistics , 1998 .

[17]  Lei Bao,et al.  Model Analysis: Representing and Assessing the Dynamics of Student Learning. , 2006 .

[18]  Jae-on Kim,et al.  Introduction to Factor Analysis , 1978 .

[19]  M. W. Richardson,et al.  The theory of the estimation of test reliability , 1937 .

[20]  F. Baker,et al.  Item response theory : parameter estimation techniques , 1993 .

[21]  R. Gorsuch.  Common Factor Analysis versus Component Analysis: Some Well and Little Known Facts. , 1990, Multivariate behavioral research.

[22]  J. Linacre,et al.  Sample size and item calibration stability , 1994 .

[23]  L. Cronbach,et al.  How we should measure "change": Or should we? , 1970 .

[24]  Robert W. Lissitz,et al.  A Suggested Change in Terminology and Emphasis Regarding Validity and Education , 2007 .

[25]  M. Aldenderfer,et al.  Cluster Analysis. Sage University Paper Series On Quantitative Applications in the Social Sciences 07-044 , 1984 .

[26]  C. Heckler.  A Step-by-Step Approach to Using the SAS™ System for Factor Analysis and Structural Equation Modeling , 1996 .

[27]  B. Kintz,et al.  Computational Handbook of Statistics , 1968 .

[28]  Frank B. Baker,et al.  Item Response Theory : Parameter Estimation Techniques, Second Edition , 2004 .

[29]  J C Loehlin,et al.  Component Analysis versus Common Factor Analysis: A Case of Disputed Authorship. , 1990, Multivariate behavioral research.

[30]  Jae-On Kim,et al.  Introduction to Factor Analysis: What It Is and How To Do It , 1978 .

[31]  Robert J. Beichner,et al.  Evaluating an electricity and magnetism assessment tool: Brief electricity and magnetism assessment , 2006 .

[32]  David Rosengrant,et al.  Multiple-choice test of energy and momentum concepts , 2003, 1602.06497.

[33]  Larry Hatcher,et al.  A Step-by-Step Approach to Using the SAS System for Factor Analysis and Structural Equation Modeling , 1994 .

[34]  Robert J. Beichner,et al.  Students’ understanding of direct current resistive electrical circuits , 2004 .

[35]  Stephen D. Baker,et al.  Testing the test: Item response curves and test quality , 2006 .

[36]  Rolph E. Anderson,et al.  Multivariate data analysis with readings (2nd ed.) , 1986 .

[37]  Susan Ramlo,et al.  Validity and reliability of the force and motion conceptual evaluation , 2008 .