Evaluation of diagnostic tools that tertiary teachers can apply to profile their students’ conceptions

ABSTRACT A multi-institution collaborative team of Australian chemistry education researchers, teaching a total of over 3000 first year chemistry students annually, has explored a tool for diagnosing students' prior conceptions as they enter tertiary chemistry courses. Five core topics were selected and clusters of diagnostic items were assembled linking related concepts in each topic together. An ordered multiple choice assessment strategy was adopted to enable provision of formative feedback to students through combination of the specific distractors that they chose. Concept items were either sourced from existing research instruments or developed by the project team. The outcome is a diagnostic tool consisting of five topic clusters of five concept items that has been delivered in large introductory chemistry classes at five Australian institutions. Statistical analysis of data has enabled exploration of the composition and validity of the instrument including a comparison between delivery of the complete 25 item instrument with subsets of five items, clustered by topic. This analysis revealed that most items retained their validity when delivered in small clusters. Tensions between the assembly, validation and delivery of diagnostic instruments for the purposes of acquiring robust psychometric research data versus their pragmatic use are considered in this study.

[1]  Jack Barbera.  A Psychometric Analysis of the Chemical Concepts Inventory , 2013 .

[2]  David F. Treagust,et al.  How Can Conceptual Change Contribute to Theory and Practice in Science Education , 2012 .

[3]  Knut Neumann,et al.  Using Ordered Multiple-Choice Items To Assess Students’ Understanding of the Structure and Composition of Matter , 2013 .

[4]  Xiaoying Xu,et al.  Understanding the State of the Art for Measurement in Chemistry Education Research: Examining the Psychometric Evidence , 2013 .

[5]  Julie Gess-Newsome,et al.  A model of teacher professional knowledge and skill including PCK: Results of the thinking from the PCK Summit , 2015 .

[6]  David F. Treagust,et al.  An Investigation into the Relationship between Students’ Conceptions of the Particulate Nature of Matter and their Understanding of Chemical Bonding , 2008 .

[7]  David F. Treagust,et al.  Evaluation of an Intervention Instructional Program to Facilitate Understanding of Basic Particle Concepts Among Students Enrolled in Several Levels of Study , 2011 .

[8]  David F. Treagust,et al.  Evaluating students' misconceptions by means of diagnostic multiple choice items , 1986 .

[9]  Stella Vosniadou,et al.  Reframing the Classical Approach to Conceptual Change: Preconceptions, Misconceptions and Synthetic Models , 2012 .

[10]  R. Coll,et al.  Understanding Mental Models of Dilution in Thai Students , 2009 .

[11]  E. Yalow.  On Educational psychology: A cognitive view. , 1979 .

[12]  Victoria Crisp,et al.  Can a picture ruin a thousand words? The effects of visual resources in exam questions , 2006 .

[13]  Stephen Krause,et al.  Development of a Chemistry Concept Inventory for Use in Chemistry, Materials and other Engineering Courses , 2004 .

[14]  Carmel McNaught,et al.  First Year on Campus: Diversity in the initial experiences of Australian undergraduates , 1995 .

[15]  Andrea A. diSessa,et al.  Knowledge in pieces : An evolving framework for understanding knowing and learning , 1988 .

[16]  R. Driver,et al.  Theories-in-Action: Some Theoretical and Empirical Issues in the Study of Students' Conceptual Frameworks in Science , 1983 .

[17]  D. Nicol,et al.  Formative assessment and self‐regulated learning: a model and seven principles of good feedback practice , 2006 .

[18]  K. Taber.  Exploring the language(s) of chemistry education , 2015 .

[19]  Susan R. Singer,et al.  Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering. , 2012 .

[20]  P. Hewson,et al.  Accommodation of a scientific conception: Toward a theory of conceptual change , 1982 .

[21]  Hans-Jürgen Schmidt.  Students' misconceptions—Looking for a pattern , 1997 .

[22]  V. Talanquer.  Central Ideas in Chemistry: An Alternative Perspective , 2016 .

[23]  Henry Heikkinen,et al.  Student ideas regarding chemical equilibrium: What written test answers do not reveal , 1990 .

[24]  Georg Rasch,et al.  Probabilistic Models for Some Intelligence and Attainment Tests , 1981, The SAGE Encyclopedia of Research Design.

[25]  A. Johnstone.  Why is science difficult to learn? Things are seldom what they seem , 1991 .

[26]  M.J. Pavelich,et al.  Development, testing, and application of a chemistry concept inventory , 2004, 34th Annual Frontiers in Education, 2004. FIE 2004..

[27]  Philip Johnson.  Children's Understanding of Changes of State Involving the Gas State, Part 1: Boiling Water and the Particle Theory. , 1998 .

[28]  Mantz Yorke,et al.  Formative Assessment and its Relevance to Retention , 2001 .

[29]  J. Gliem,et al.  Calculating, Interpreting, And Reporting Cronbach’s Alpha Reliability Coefficient For Likert-Type Scales , 2003 .

[30]  T. Haladyna,et al.  Validity of a Taxonomy of Multiple-Choice Item-Writing Rules. , 1989 .

[31]  Catherine M. Scott,et al.  Understanding Mental Models of Dilution in Thai Students , 2014 .

[32]  Alipaşa Ayas,et al.  Promoting conceptual change in first year students’ understanding of evaporation , 2010 .

[33]  Andrea A. diSessa,et al.  A bird's-eye view of the "pieces" vs "coherence" controversy (from the "pieces" side of the fence") , 2013 .

[34]  Victor M. S. Gil,et al.  The Complexity of Teaching and Learning Chemical Equilibrium , 2000 .

[35]  P. O’Toole,et al.  A Picture is Worth a Thousand Words: Investigating First Year Chemistry Students’ Ability to Visually Express Their Understanding of Chemistry Concepts , 2016 .

[36]  Ellen J. Yezierski,et al.  Misconceptions about the Particulate Nature of Matter. Using Animations To Close the Gender Gap , 2006 .

[37]  M. Linn,et al.  Heat energy and temperature concepts of adolescents, adults, and experts: Implications for curricular improvements , 1994 .

[38]  Russell Tytler,et al.  A comparison of year 1 and year 6 students' conceptions of evaporation and condensation: dimensions of conceptual progression , 2000 .

[39]  W. Robinson,et al.  An Inventory for Alternate Conceptions among First-Semester General Chemistry Students , 2002 .

[40]  Michael C. Rodriguez,et al.  A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment , 2002 .

[41]  M. Potgieter,et al.  Assessment of preparedness of first-year chemistry students: development and application of an instrument for diagnostic and placement purposes , 2008 .

[42]  Gabriela C. Weaver,et al.  Transforming Institutions: Undergraduate STEM Education for the 21st Century , 2015 .

[43]  Wendy K. Adams,et al.  Development and Validation of Instruments to Measure Learning of Expert‐Like Thinking , 2011 .

[44]  Stacey Lowery Bretz,et al.  Development of the Bonding Representations Inventory to Identify Student Misconceptions about Covalent and Ionic Bonding Representations. , 2014 .

[45]  M. Potgieter,et al.  Preparedness for tertiary chemistry: multiple applications of the chemistry competence test for diagnostic and prediction purposes , 2011 .

[46]  David F. Treagust,et al.  Development and Application of a Diagnostic Instrument to Evaluate Grade-11 and -12 Students' Concepts of Covalent Bonding and Structure Following a Course of Instruction. , 1989 .

[47]  J. Reilly,et al.  You can't get there from here , 2016, Catheterization and cardiovascular interventions : official journal of the Society for Cardiac Angiography & Interventions.

[48]  Ajda Kahveci.  Diagnostic Assessment of Student Understanding of the Particulate Nature of Matter: Decades of Research , 2013 .

[49]  Klaas Sijtsma,et al.  On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s Alpha , 2008, Psychometrika.

[50]  A. H. Johnstone,et al.  You Can’t Get There from Here1 , 2010 .

[51]  Roy Tasker.  Research into practice: Visualising the molecular world for a deep understanding of chemistry , 2014 .

[52]  Saša A. Glažar,et al.  The development of the concept of ‘matter’: a cross‐age study of how children describe materials , 2005 .

[53]  Jack Barbera,et al.  Gathering Evidence for Validity during the Design, Development, and Qualitative Evaluation of Thermochemistry Concept Inventory Items , 2013 .

[54]  D. Treagust,et al.  Conceptual change: A powerful framework for improving science teaching and learning , 2003 .

[55]  Ellen J. Yezierski,et al.  Evidence for the effectiveness of inquiry-based, particulate-level instruction on conceptions of the particulate nature of matter , 2012 .

[56]  Derek C. Briggs,et al.  Diagnostic Assessment With Ordered Multiple-Choice Items , 2006 .

[57]  M. Tavakol,et al.  Making sense of Cronbach's alpha , 2011, International journal of medical education.

[58]  Patrick J. Garnett,et al.  Misconceptions of chemical equilibrium , 1985 .

[59]  From Inert Object to Chemical Substance: Students' Initial Conceptions and Conceptual Development during an Introductory Experimental Chemistry Sequence. , 2000 .

[60]  Stacey Lowery Bretz,et al.  The Development of the Redox Concept Inventory as a Measure of Students' Symbolic and Particulate Redox Understandings and Confidence. , 2014 .

[61]  David F. Treagust,et al.  Evaluation of Students’ Understanding of Thermal Concepts in Everyday Contexts , 2012 .

[62]  Roger Osborne,et al.  Children's conceptions of the changes of state of water , 1983 .

[63]  Vicente A Talanquer,et al.  How Do Students Reason About Chemical Substances and Reactions , 2013 .

[64]  David F. Treagust,et al.  EVALUATING STUDENTS’ UNDERSTANDING OF KINETIC PARTICLE THEORY CONCEPTS RELATING TO THE STATES OF MATTER, CHANGES OF STATE AND DIFFUSION: A CROSS-NATIONAL STUDY , 2010 .

[65]  Roy Tasker,et al.  Research into practice: visualisation of the molecular world using animations , 2006 .

[66]  Vicente A Talanquer,et al.  Explanations and Teleology in Chemistry Education , 2007 .

[67]  Andrea A. diSessa,et al.  Coherence versus fragmentation in the development of the concept of force , 2004, Cogn. Sci..

[68]  Kristen L. Murphy,et al.  Building the ACS Exams Anchoring Concept Content Map for Undergraduate Chemistry , 2012 .

[69]  Zubeyde Demet Kirbulut,et al.  Consistency of Students’ Ideas across Evaporation, Condensation, and Boiling , 2013 .

[70]  G. Lawrie,et al.  Insights and Teacher Perceptions Regarding Students’ Conceptions as They Enter Tertiary Chemistry Studies: a Comparative Study , 2018 .

[71]  E. Wiebe,et al.  The influence of prior knowledge on viewing and interpreting graphics with macroscopic and molecular representations , 2008 .

[72]  Assessing Secondary and College Students’ Implicit Assumptions about the Particulate Nature of Matter: Development and Validation of the Structure and Motion of Matter Survey , 2011 .

[73]  George M. Bodner,et al.  I HAVE FOUND YOU AN ARGUMENT: THE CONCEPTUAL KNOWLEDGE OF BEGINNING CHEMISTRY GRADUATE STUDENTS , 1991 .

[74]  Simon B Bedford,et al.  Closing the loop: A model for inter-institutional collaboration through delivering formative assessment in large, first-year STEM classes , 2016 .

[75]  K. Berg.  A study of first-year chemistry students' understanding of solution concentration at the tertiary level , 2012 .

[76]  Jack Barbera,et al.  Psychometric analysis of the thermochemistry concept inventory , 2014 .

[77]  P. Schwartz,et al.  Evaluating the Content and Response Process Validity of Data from the Chemical Concepts Inventory , 2014 .

[78]  Margot A. Vigeant,et al.  Development of the Heat and Energy Concept Inventory: Preliminary Results on the Prevalence and Persistence of Engineering Students' Misconceptions , 2012 .

[79]  Gwendolyn A. Lawrie,et al.  Development of scaffolded online modules to support self-regulated learning in chemistry concepts , 2016 .

[80]  Iztok Devetak,et al.  Assessing 16-Year-Old Students’ Understanding of Aqueous Solution at Submicroscopic Level , 2009 .

[81]  G. Solano-Flores,et al.  On the cultural validity of science assessments , 2001 .

[82]  Simon B Bedford,et al.  Using formative feedback to identify and support first-year chemistry students with missing or misconceptions. A Practice Report , 2013 .

[83]  S. Vosniadou.  The Framework Theory Approach to the Problem of Conceptual Change , 2009 .

[84]  D. Treagust.  Development and use of diagnostic tests to evaluate students’ misconceptions in science , 1988 .

[85]  Alex H. Johnstone,et al.  Chemical education research: Facts, findings, and consequences , 1983 .

[86]  David F. Treagust,et al.  Application of a conceptual conflict teaching strategy to enhance student learning of acids and bases , 1988 .

[87]  J. Biggs.  What do inventories of students' learning processes really measure? A theoretical review and clarification. , 1993, The British journal of educational psychology.