Understanding the State of the Art for Measurement in Chemistry Education Research: Examining the Psychometric Evidence

Many of the instruments developed for research use by the chemistry education community are relatively new. Because psychometric evidence dictates the validity of interpretations made from test scores, gathering and reporting validity and reliability evidence is of utmost importance. Therefore, the purpose of this study was to investigate what “counts” as psychometric evidence within this community. Using a methodology based on concepts described and operationalized in the Standards for Educational and Psychological Testing, instruments first published in the Journal between 2002 and 2011, and follow-up publications reporting the use of these instruments, were examined. Specifically, we investigated the availability of evidence based on test content, response processes, internal structure, relations to other variables, temporal stability, and internal consistency. Findings suggest that our peer review and reporting practices value some types of evidence while neglecting others. Results of this study serve...

[1]  Robert F. Dedrick,et al.  Hierarchical Confirmatory Factor Analysis of the Child Behavior Checklist/4-18. , 1998 .

[2]  Zuzana Haláková,et al.  Two Kinds of Conceptual Problems in Chemistry Teaching , 2007 .

[3]  Stacey Lowery Bretz,et al.  Diagnosing changes in attitude in first-year college chemistry students with a shortened version of Bauer's semantic differential , 2011 .

[4]  Diane M. Bunce,et al.  Report of the Task Force on Chemical Education Research of the American Chemical Society Division of Chemical Education , 1994 .

[5]  Carol M. Woods.  Ramsay-Curve Differential Item Functioning , 2011 .

[6]  J. Lewis,et al.  Refinement of a Chemistry Attitude Measure for College Students. , 2011 .

[7]  Hannah Sevian,et al.  Incrementally Approaching an Inquiry Lab Curriculum: Can Changing a Single Laboratory Experiment Improve Student Performance in General Chemistry?. , 2009 .

[8]  Diane M. Bunce,et al.  Research in Chemical Education - the Third Branch of Our Profession , 1997 .

[9]  Xiaoying Xu,et al.  Comparing Two Tests of Formal Reasoning in a College Chemistry Context , 2010 .

[10]  K. L. Murphy,et al.  Investigating Factors That Influence Item Performance on ACS Exams , 2012 .

[11]  J. Lewis,et al.  A Psychometric Evaluation of the Colorado Learning Attitudes about Science Survey for Use in Chemistry , 2012 .

[12]  Klaas Sijtsma,et al.  On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s Alpha , 2008, Psychometrika.

[13]  Katerina Salta,et al.  Attitudes toward chemistry among 11th grade students in high schools in Greece , 2004 .

[14]  Michael Miller.  Coefficient alpha: A basic introduction from the perspectives of classical test theory and structural equation modeling , 1995 .

[15]  Scott E. Lewis,et al.  Attitude Counts: Self-Concept and Success in General Chemistry , 2009 .

[16]  Vickie M. Williamson,et al.  Surveying Students' Attitudes and Perceptions toward Guided-Inquiry and Open-Inquiry Laboratories , 2009 .

[17]  K. Knaus,et al.  A Valid and Reliable Instrument for Cognitive Complexity Rating Assignment of Chemistry Exam Items , 2011 .

[18]  S. Embretson.  A general latent trait model for response processes , 1984 .

[19]  Derek Cheung,et al.  Teacher Beliefs about Implementing Guided-Inquiry Laboratory Experiments for Secondary School Chemistry , 2011 .

[20]  Assessing Secondary and College Students’ Implicit Assumptions about the Particulate Nature of Matter: Development and Validation of the Structure and Motion of Matter Survey , 2011 .

[21]  Paul J. Germann,et al.  Development of the attitude toward science in school assessment and its use to investigate the relationship between science achievement and attitude toward science in school , 1988 .

[22]  W. Robinson,et al.  An Inventory for Alternate Conceptions among First-Semester General Chemistry Students , 2002 .

[23]  J. Dudley Herron,et al.  Chemical Education Research: Improving Chemistry Learning. , 1999 .

[24]  Jill M. Aldridge,et al.  Development and Validation of an Instrument to Measure Students’ Motivation and Self‐Regulation in Science Learning , 2011 .

[25]  S. Messick.  Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[26]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[27]  M. Kane.  Current Concerns in Validity Theory , 2001 .

[28]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.

[29]  K. Oloruntegbe,et al.  Ecocultural Factors in Students' Ability To Relate Science Concepts Learned at School and Experienced at Home: Implications for Chemistry Education , 2011 .

[30]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[31]  Michael T. Kane,et al.  An argument-based approach to validity. , 1992 .

[32]  Alipaşa Ayas,et al.  Promoting conceptual change in first year students’ understanding of evaporation , 2010 .

[33]  R. Peterson.  A Meta-analysis of Cronbach's Coefficient Alpha , 1994 .

[34]  Jennifer Loertscher,et al.  Development and analysis of an instrument to assess student understanding of foundational concepts before biochemistry coursework * , 2011, Biochemistry and molecular biology education : a bimonthly publication of the International Union of Biochemistry and Molecular Biology.

[35]  Regina F. Frey,et al.  Peer-Led Team Learning in General Chemistry: Implementation and Evaluation , 2008 .

[36]  C. Bauer.  Beyond "Student Attitudes": Chemistry Self-Concept Inventory for Assessment of the Affective Component of Student Learning , 2005 .

[37]  Joanna S. Gorin.  Reconsidering Issues in Validity Theory , 2007 .

[38]  Laura M. Desimone,et al.  Are We Asking the Right Questions? Using Cognitive Interviews to Improve Surveys in Education Research , 2004 .

[39]  Katerina Salta,et al.  Exploring Novel Tools for Assessing High School Students' Meaningful Understanding of Organic Reactions , 2011 .

[40]  Peter E. Childs,et al.  The use of an intervention programme to improve undergraduatestudents' chemical knowledge and address their misconceptions , 2011 .

[41]  Karen A. Gibbs,et al.  Discovering the Determinants of Chemistry Course Perceptions in Undergraduate Students , 2010 .

[42]  Zuhao Wang,et al.  Using Rasch Measurement to Develop a Computer Modeling-Based Instrument to Assess Students' Conceptual Understanding of Matter , 2012 .

[43]  M. Potgieter,et al.  Preparedness for tertiary chemistry: multiple applications of the chemistry competence test for diagnostic and prediction purposes , 2011 .

[44]  A. Özkaya,et al.  Effectiveness of Conceptual Change-Oriented Teaching Strategy To Improve Students' Understanding of Galvanic Cells , 2006 .

[45]  Lizelle Fletcher,et al.  Inaccuracy of self-evaluation as additional variable for prediction of students at risk of failing first-year chemistry , 2010 .

[46]  Andrea Gay Duzor.  Capitalizing on Teacher Expertise: Motivations for Contemplating Transfer from Professional Development to the Classroom , 2011 .

[47]  Stacey Lowery Bretz,et al.  CHEMX: An Instrument To Assess Students' Cognitive Expectations for Learning Chemistry , 2007 .

[48]  Melanie M. Cooper,et al.  Design and Validation of an Instrument To Assess Metacognitive Skillfulness in Chemistry Problem Solving , 2009 .

[49]  S. Messick.  Meaning and Values in Test Validation: The Science and Ethics of Assessment , 1989 .

[50]  P. Bell,et al.  Knowledge Surveys in General Chemistry: Confidence, Overconfidence, and Performance , 2011 .

[51]  Thomas W. Conkling,et al.  Google Scholar’s Coverage of the Engineering Literature: An Empirical Study , 2008 .

[52]  Tao Hong,et al.  A Psychometric Re‐Evaluation of the Design, Engineering and Technology (DET) Survey , 2011 .

[53]  Pamela A. Moss,et al.  Can There Be Validity Without Reliability? , 1994 .

[54]  Christopher F. Bauer,et al.  Attitude toward Chemistry: A Semantic Differential Instrument for Assessing Curriculum Impacts , 2008 .

[55]  Ronald H. Stevens,et al.  Reliable multi method assessment of metacognition use in chemistry problem solving , 2008 .

[56]  Katherine K. Perkins,et al.  Modifying and Validating the Colorado Learning Attitudes about Science Survey for Use in Chemistry , 2008 .

[57]  D. Kember,et al.  The dimensionality of approaches to learning : an investigation with confirmatory factor analysis on the structure of the SPQ and LPQ , 1998 .

[58]  H. Walberg,et al.  A structural model of science achievement and attitude : an extension to high school , 1992 .

[59]  Saša A. Glažar,et al.  Intrinsic Motivation of Pre‐service Primary School Teachers for Learning Chemistry in Relation to their Academic Achievement , 2008 .