Automated summary evaluation with inbuilt rubric method: An alternative to constructed responses and multiple-choice tests assessments

Abstract Automated summary evaluation is proposed as an alternative to rubrics and multiple-choice tests in knowledge assessment. Inbuilt rubric is a recent Latent Semantic Analysis (LSA) method that implements rubrics in an artificially-generated semantic space. It was compared with classical LSA’s cosine-based methods assessing knowledge in a within-subjects design regarding two validation sources: a comparison with the results of rubric scores and multiple-choice tests, and the sensitivity of predicting the academic level of the test-taker. Results showed a higher reliability for inbuilt rubric (from Pearson correlation coefficient .81 to .49) over the classical LSA method (from .61 to .34), and a higher sensitivity using binary logistic regressions and effect sizes to predict academic level. It is concluded that inbuilt rubric has a qualitatively higher reliability and validity than classical LSA methods in a way that is complementary to models based on semantic networks. Thus, it is concluded that new automated summary evaluation approaches such as the inbuilt rubric method can be practical in terms of reliability and efficiency, and, thus, they can offer an affordable and valuable form of knowledge assessment in different educational levels.

[1]  Paul A. Kirschner,et al.  Evaluating assessment quality in competence-based education: A qualitative comparison of two frameworks , 2007 .

[2]  Gerry Stahl,et al.  Developing Summarization Skills through the Use of LSA-Based Feedback , 2000, Interact. Learn. Environ..

[3]  Ricardo Olmos,et al.  Transforming Selected Concepts Into Dimensions in Latent Semantic Analysis , 2014 .

[4]  Klaus Zechner,et al.  Automated Essay Scoring: Writing Assessment and Instruction , 2010 .

[5]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[6]  G. A. Marcoulides,et al.  An Introduction to Applied Multivariate Analysis , 2008 .

[7]  Robert J. Mislevy,et al.  Automated Scoring with Validity in Mind , 2016 .

[8]  Danielle S. McNamara,et al.  The Use of Latent Semantic Analysis as a Tool for the Quantitative Assessment of Understanding and Knowledge , 2000 .

[9]  H. Andrade,et al.  A review of rubric use in higher education , 2010 .

[10]  Patrick F. Reidy.  An Introduction to Latent Semantic Analysis , 2009 .

[11]  Phillip Dawson,et al.  Assessment rubrics: towards clearer and more replicable design, research and practice , 2017 .

[12]  Danielle S. McNamara,et al.  Handbook of latent semantic analysis , 2007 .

[13]  José J. Cañas,et al.  Assessing short summaries with human judgments procedure and latent semantic analysis in narrative and expository texts , 2006, Behavior research methods.

[14]  Angelo Kyrilov,et al.  Automated assessment of short free-text responses in computer science using latent semantic analysis , 2011, ITiCSE '11.

[15]  Mariola Moeyaert,et al.  Effects of a Summarizing Strategy on Written Summaries of Children With Emotional and Behavioral Disorders , 2017 .

[16]  Tami Seifert,et al.  Online self-assessment and peer-assessment as a tool to enhance student-teachers’ assessment skills , 2018, Assessment & Evaluation in Higher Education.

[17]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[18]  W. Kintsch,et al.  Strategies of discourse comprehension , 1983 .

[19]  Anders Jonsson,et al.  The use of scoring rubrics: Reliability, validity, and educational consequences , 2007 .

[20]  Jon A. Willits,et al.  Models of Semantic Memory , 2015 .

[21]  David Boud,et al.  What makes for effective feedback: staff and student perspectives , 2019 .

[22]  Danielle S. McNamara,et al.  Assessing and Improving Comprehension With Latent Semantic Analysis , 2007 .

[23]  R. McGrath,et al.  When effect sizes disagree: the case of r and d. , 2006, Psychological methods.

[24]  Michael Mogessie Ashenafi.  Peer-assessment in higher education – twenty-first century practices, challenges and the way forward , 2017 .

[25]  Guillermo Jorge-Botana.  Gallito 2 . 0 : a Natural Language Processing tool to support Research on Discourse , 2013 .

[26]  José A. Martínez-Huertas,et al.  Analyzing Two Automatic Latent Semantic Analysis (LSA) Assessment Methods (Inbuilt Rubric vs. Golden Summary) in Summaries Extracted from Expository Texts , 2018 .

[27]  Danielle S. McNamara,et al.  Computational Methods to Extract Meaning From Text and Advance Theories of Human Cognition , 2011, Top. Cogn. Sci..

[28]  Arthur C. Graesser,et al.  Strengths, Limitations, and Extensions of LSA , 2007 .

[29]  Anders Jonsson,et al.  The Use of Scoring Rubrics for Formative Assessment Purposes Revisited: A Review. , 2013 .

[30]  S. Vaughn,et al.  A Review of Summarizing and Main Idea Interventions for Struggling Readers in Grades 3 Through 12: 1978–2016 , 2019 .

[31]  Peter W. Foltz,et al.  Automated Essay Scoring: Applications to Educational Technology , 1999 .

[32]  Scott Dooley,et al.  Summary Street®: Computer Support for Comprehension and Writing , 2005 .

[33]  Yao-Ting Sung,et al.  The effect of online summary assessment and feedback system on the summary writing on 6th graders: The LSA-based technique , 2016, Comput. Educ..

[34]  Ricardo Olmos,et al.  Transforming LSA space dimensions into a rubric for an automatic assessment and feedback system , 2016, Inf. Process. Manag..

[35]  Eileen Kintsch,et al.  Summary Street: Interactive Computer Support for Writing , 2004 .

[36]  Adelaide V. Finch,et al.  September , 1867, The Hospital.