EvoGrader: an online formative assessment tool for automatically evaluating written evolutionary explanations

EvoGrader is a free, online, on-demand formative assessment service designed for use in undergraduate biology classrooms. EvoGrader’s web portal is powered by Amazon’s Elastic Cloud and run with LightSIDE Lab’s open-source machine-learning tools. The EvoGrader web portal allows biology instructors to upload a response file (.csv) containing unlimited numbers of evolutionary explanations written in response to 86 different ACORNS (Assessing COntextual Reasoning about Natural Selection) instrument items. The system automatically analyzes the responses and provides detailed information about the scientific and naive concepts contained within each student’s response, as well as overall student (and sample) reasoning model types. Graphs and visual models provided by EvoGrader summarize class-level responses; downloadable files of raw scores (in .csv format) are also provided for more detailed analyses. Although the computational machinery that EvoGrader employs is complex, using the system is easy. Users only need to know how to use spreadsheets to organize student responses, upload files to the web, and use a web browser. A series of experiments using new samples of 2,200 written evolutionary explanations demonstrate that EvoGrader scores are comparable to those of trained human raters, although EvoGrader scoring takes 99% less time and is free. EvoGrader will be of interest to biology instructors teaching large classes who seek to emphasize scientific practices such as generating scientific explanations, and to teach crosscutting ideas such as evolution and natural selection. The software architecture of EvoGrader is described as it may serve as a template for developing machine-learning portals for other core concepts within biology and across other disciplines.

[1]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .

[2]  Zellig S. Harris,et al.  Distributional Structure , 1954 .

[3]  Louise T. Su.  The Relevance of Recall and Precision in User Evaluation , 1994, J. Am. Soc. Inf. Sci..

[4]  Charles W. Anderson,et al.  Student conceptions of natural selection and its role in evolution , 1986 .

[5]  Mark Urban-Lurain,et al.  What Are They Thinking? Automated Analysis of Student Writing about Acid–Base Chemistry in Introductory Biology , 2012, CBE life sciences education.

[6]  Minsu Ha,et al.  Item Feature Effects in Evolution Assessment. , 2011 .

[7]  Yoshihiro Yamanishi,et al.  The inference of protein-protein interactions by co-evolutionary analysis is improved by excluding the information about the phylogenetic relationships , 2005, Bioinform..

[8]  Helen R. Quinn,et al.  A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas , 2013 .

[9]  Zellig S. Harris,et al.  Distributional Structure , 1954 .

[10]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[11]  Isaac I. Bejar,et al.  A methodology for scoring open-ended architectural design problems. , 1991 .

[12]  Ross H. Nehm,et al.  Human vs. Computer Diagnosis of Students’ Natural Selection Knowledge: Testing the Efficacy of Text Analytic Software , 2011, Journal of Science Education and Technology.

[13]  Ross H. Nehm,et al.  The future of natural selection knowledge measurement: A reply to Anderson et al. (2010) , 2009 .

[14]  David A. Gillam,et al.  A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas , 2012 .

[15]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[16]  Fabrice Muhlenbach,et al.  Identifying and Handling Mislabelled Instances , 2004, Journal of Intelligent Information Systems.

[17]  Elijah Mayfield,et al.  Transforming Biology Assessment with Machine Learning: Automated Scoring of Written Evolutionary Explanations , 2012 .

[18]  K. Sheppard,et al.  Academic preparation in biology and advocacy for teaching evolution: Biology versus non-biology teachers , 2009 .

[19]  G. Church,et al.  Computational identification of transcription factor binding sites via a transcription-factor-centric clustering (TFCC) algorithm. , 2002, Journal of molecular biology.

[20]  A. Duncan.  Why we need high-speed schools. , 2013, Scientific American.

[21]  R. Nehm,et al.  Biology Majors' Knowledge and Misconceptions of Natural Selection , 2007 .

[22]  W. B. Cavnar,et al.  N-gram-based text categorization , 1994 .

[23]  Mark Urban-Lurain,et al.  Applying Computerized-Scoring Models of Written Biological Explanations across Courses and Colleges: Prospects and Limitations , 2011, CBE life sciences education.

[24]  Ron Good,et al.  Students' conceptions of natural selection and its role in evolution: Cases of replication and comparison , 1995 .

[25]  Minsu Ha,et al.  Reasoning About Natural Selection: Diagnosing Contextual Competency using the ACORNS Instrument , 2012 .

[26]  J. Opfer,et al.  Cognitive foundations for science assessment design: Knowing what students know about evolution , 2012 .

[27]  W. L. Eikenberry.  The national association for research in science teaching , 1929 .

[28]  Jimmy J. Lin,et al.  Document vector representations for feature extraction in multi-stage document ranking , 2013, Information Retrieval.

[29]  Ross H. Nehm,et al.  Measuring knowledge of natural selection: A comparison of the CINS, an open‐response instrument, and an oral interview , 2008 .

[30]  Minsu Ha,et al.  Assessing Scientific Practices Using Machine-Learning Methods: How Closely Do They Match Clinical Interview Performance? , 2013, Journal of Science Education and Technology.