Measuring knowledge of natural selection: A comparison of the CINS, an open‐response instrument, and an oral interview

Growing recognition of the central importance of fostering an in-depth understanding of natural selection has, surprisingly, failed to stimulate work on the development and rigorous evaluation of instruments that measure knowledge of it. We used three different methodological tools, the Conceptual Inventory of Natural Selection (CINS), a modified version of Bishop and Anderson's (Bishop and Anderson [1990] Journal of Research in Science Teaching 27: 415–427) open-response test that we call the Open Response Instrument (ORI), and an oral interview derived from both instruments, to measure biology majors' understanding of and alternative conceptions about natural selection. We explored how these instruments differentially inform science educators about the knowledge and alternative conceptions their students harbor. Overall, both the CINS and ORI provided excellent replacements for the time-consuming process of oral interviews and produced comparable measures of key concept diversity and, to a lesser extent, key concept frequency. In contrast, the ORI and CINS produced significantly different measures of both alternative conception diversity and frequency, with the ORI results completely concordant with oral interview results. Our study indicated that revisions of both the CINS and ORI are necessary because of numerous instrument items characterized by low discriminability, high and/or overlapping difficulty, and mismatches with the sample. While our results revealed that both instruments are valid and generally reliable measures of knowledge and alternative conceptions about natural selection, a test combining particular components of both instruments—a modified version of the CINS to test for key concepts, and a modified version of the ORI to assess student alternative conceptions—should be used until a more approprite instrument is developed and rigorously evaluated. © 2008 Wiley Periodicals, Inc. J Res Sci Teach 45: 1131–1160, 2008

[1]  J. Millman,et al.  THE ISSUE OF ITEM AND TEST VARIANCE FOR CRITERION-REFERENCED TESTS: A CLARIFICATION , 1974 .

[2]  M. Zimmerman,et al.  The Evolution-Creation Controversy: Opinions of Ohio High School Biology Teachers , 1987 .

[3]  Michael C. Rodriguez.  Construct Equivalence of Multiple-Choice and Constructed-Response Items: A Random Effects Synthesis of Correlations , 2003 .

[4]  Lawrence C. Scharmann,et al.  Teaching evolution: Understanding and applying the nature of science , 1992 .

[5]  James W. Pellegrino,et al.  Knowing What Students Know. , 2003 .

[6]  Zoubeida R. Dagher,et al.  Scientific views and religious beliefs of college students: The case of biological evolution , 1997 .

[7]  Robert Lukhele,et al.  On the Relative Value of Multiple-Choice, Constructed Response, and Examinee-Selected Items on Two Achievement Tests. Program Statistics Research Technical Report No. 93-28. , 1993 .

[8]  Ron Good,et al.  Students' conceptions of natural selection and its role in evolution: Cases of replication and comparison , 1995 .

[9]  Howard Wainer,et al.  ON THE RELATIVE VALUE OF MULTIPLE‐CHOICE, CONSTRUCTED‐RESPONSE, AND EXAMINEE‐SELECTED ITEMS ON TWO ACHIEVEMENT TESTS1 , 1993 .

[10]  Anton Béguin.  Applications of Rasch Measurement in Science Education , 2009 .

[11]  Michelle D. Young,et al.  Coloring Epistemologies: Are Our Research Epistemologies Racially Biased? , 1997 .

[12]  L. S. Lerner.  Good Science, Bad Science: Teaching Evolution in the States , 2000 .

[13]  Mark G. Simkin,et al.  How Well Do Multiple Choice Tests Evaluate Student Understanding in Computer Programming Classes? , 2003, J. Inf. Syst. Educ..

[14]  dkk Donald Ary.  Introduction to research in education , 1972 .

[15]  Ross H. Nehm,et al.  Does Increasing Biology Teacher Knowledge of Evolution and the Nature of Science Lead to Greater Preference for the Teaching of Evolution in Schools? , 2007 .

[16]  C. Chou,et al.  A NATIONAL SURVEY OF STUDENTS ’ CONCEPTIONS IN CHEMISTRY IN TAIWAN , 2005 .

[17]  C. Fox,et al.  Applying the Rasch Model: Fundamental Measurement in the Human Sciences , 2001 .

[18]  Lawrence C. Scharmann.  Teaching evolution: Designing successful instruction , 1993 .

[19]  Rebecca Schneider.  Science Teacher Educators as a Community of Practice , 2007 .

[20]  R. Nehm,et al.  Biology Majors' Knowledge and Misconceptions of Natural Selection , 2007 .

[21]  G. Norman,et al.  Development and Evaluation of the Conceptual Inventory of Natural Selection , 2002 .

[22]  G. Skoog,et al.  The emphasis given to evolution in state science standards: A lever for change in evolution education? , 2002 .

[23]  A. Anastasi.  Psychological testing, 4th ed. , 1976 .

[24]  Kathryn Scantlebury,et al.  The role of Rasch analysis when conducting science education research utilizing multiple‐choice tests , 2006 .

[25]  J. Maienschein.  Growth of biological thought , 1994, Nature.

[26]  L. Cronbach.  Dissent from Carver. , 1975 .

[27]  L. Henriques,et al.  Clergy views on evolution, creationism, science, and religion , 2006 .

[28]  Benjamin D. Wright,et al.  Rasch Measurement in Higher Education , 1999 .

[29]  W. Popham.  Assessment for educational leaders / W. James Popham , 2006 .

[30]  Ronald P. Carver,et al.  Two Dimensions of Tests: Psychometric and Edumetric. , 1974 .

[31]  Sherry A. Southerland,et al.  Intentions and beliefs in students' understanding and acceptance of biological evolution , 2003 .

[32]  Randy Moore.  Teaching Evolution: Do State Standards Matter? , 2002 .

[33]  Ronald D. Simpson,et al.  Attitudes of introductory college biology students toward evolution , 1982 .

[34]  Xiufeng Liu,et al.  Students' progression of understanding the matter concept from elementary to high school , 2005 .

[35]  Lisa A. Donnelly,et al.  Biology Teachers' Attitudes toward and Use of Indiana's Evolution Standards. , 2007 .

[36]  Charles W. Anderson,et al.  Student conceptions of natural selection and its role in evolution , 1986 .

[37]  Philip Sedgwick,et al.  Control groups , 2010, BMJ : British Medical Journal.

[38]  J. Shea.  National Science Education Standards , 1995 .

[39]  M. Zeidner.  Essay Versus Multiple-Choice Type Classroom Exams: The Student's Perspective. , 1987 .

[40]  Téléphone,et al.  THE ONTARIO INSTITUTE FOR STUDIES IN EDUCATION , 2007 .

[41]  Learning about biological evolution: A special case of intentional conceptual change , 2003 .

[42]  S. Gould.  The Structure of Evolutionary Theory , 2002 .

[43]  Elizabeth C. Doster,et al.  Hearts and minds in the science classroom: The education of a confirmed evolutionist , 1995 .

[44]  R. Nehm,et al.  Faith-based Evolution Education? , 2006 .

[45]  Elizabeth Engel Clough,et al.  How Secondary Students Interpret Instances of Biological Adaptation. , 1985 .

[46]  B. Bloom,et al.  Taxonomy of Educational Objectives. Handbook I: Cognitive Domain , 1966 .

[47]  M. Brumby,et al.  Misconceptions about the concept of natural selection by medical biology students , 1984 .

[48]  Ö. Geban,et al.  Undergraduate Pre-Service Teachers' Understandings and Misconceptions of Phase Equilibrium , 2006 .

[49]  B. Crawford,et al.  Confronting Prospective Teachers' Ideas of Evolution and Scientific Inquiry Using Technology and Inquiry-Based Tasks , 2005 .

[50]  Brent Bridgeman,et al.  A Comparison of Quantitative Questions in Open‐Ended and Multiple‐Choice Formats , 1992 .

[51]  Ella L. Ingram,et al.  Relationship between Achievement and Students' Acceptance of Evolution or Creation in an Upper-Level Evolution Course. , 2006 .