Modeling Holistic Marks With Analytic Rubrics

Analytic and holistic marking are typically researched as opposites, generating a mixed and inconclusive evidence base. Holistic marking is low on content validity but efficient. Analytic approaches are praised for transparency and detailed feedback. Capturing complex criteria interactions, when deciding marks, is claimed to be better suited to holistic approaches whilst analytic rules are thought to be limited. Both guidance and evidence in this area remain limited to date. Drawing from the known complementary strengths of these approaches, a university department enhanced its customary holistic marking practices by introducing analytic rubrics for feedback and as ancillary during marking. The customary holistic approach to deciding marks was retained in the absence of a clear rationale from the literature. Exploring the relationship between the analytic criteria and holistic marks became the focus of an exploratory study during a trial year that would use two perspectives. Following guidance from the literature, practitioners formulated analytic rules drawing on their understanding of the role of criteria, to explain output marks by allocating weightings. Secondly, data derived throughout the year consisting of holistic marks and analytic judgements (criteria) was analysed using machine learning techniques (random forests). This study reports on data from essay-based questions (exams) for years 2 and 3 of study (n = 3,436). Random forests provided a ranking of the variable importance of criteria relative to holistic marks, which was used to create criterion weightings (data-derived). Moreover, illustrative decision trees provide insights into non-linear roles of criteria for different levels of achievement. Criterion weightings, expected by practitioners and data-derived (from holistic marks), reveal contrasts in the ranking of top criteria within and across years. Our exploratory study confirms that holistic and analytic approaches, combined, offer promising and productive ways forward both in research and practice to gain insight into the nature of overall marks and relations with criteria. Rather than opposites, these approaches offer complementary insights to help substantiate claims made in favour of holistic marking. Our findings show that analytic may offer insights into the extent to which holistic marking really aligns with assumptions made. Limitations and further investigations are discussed.

[1]  G. Boulton‐Lewis.  Teaching for quality learning at university , 2008 .

[2]  S. Messick.  The Interplay of Evidence and Consequences in the Validation of Performance Assessments , 1994 .

[3]  What determines GCSE marking accuracy? An exploration of expertise among maths and physics markers , 2008 .

[4]  F. Prins,et al.  Editorial: Transparency in Assessment—Exploring the Influence of Explicit Assessment Criteria , 2019, Front. Educ..

[5]  Mantz Yorke,et al.  Summative assessment: dealing with the ‘measurement fallacy’ , 2011 .

[6]  H. Andrade,et al.  A review of rubric use in higher education , 2010 .

[7]  P. Grainger,et al.  Judging quality through substantive conversations between markers , 2008 .

[8]  Understanding self-regulated learning in open-ended online assignment tasks , 2018 .

[9]  Phillip Dawson,et al.  Assessment rubrics: towards clearer and more replicable design, research and practice , 2017 .

[10]  J. Friedman.  Greedy function approximation: A gradient boosting machine. , 2001 .

[11]  D. Royce Sadler,et al.  Indeterminacy in the use of preset criteria for assessment and grading , 2009 .

[12]  D. Royce Sadler,et al.  Specifying and Promulgating Achievement Standards , 1987 .

[13]  Susan M. Brookhart,et al.  The quality and effectiveness of descriptive rubrics , 2015 .

[14]  F. Dochy.  The Edumetric Quality of New Modes of Assessment: Some Issues and Prospects , 2009 .

[15]  B. Huot,et al.  Reliability, Validity, and Holistic Scoring: What We Know and What We Need to Know , 1990 .

[16]  Susan M. Brookhart,et al.  Appropriate Criteria: Key to Effective Rubrics , 2018, Front. Educ..

[17]  Lenore Adie,et al.  What’s the point of moderation? A discussion of the purposes achieved through contemporary moderation practices , 2016 .

[18]  H. Andrade.  Teaching With Rubrics: The Good, the Bad, and the Ugly , 2005 .

[19]  M. Price,et al.  External examining: fit for purpose? , 2015 .

[20]  D. Macdonald,et al.  (Mis)appropriations of criteria and standards‐referenced assessment in a performance‐based subject , 2008 .

[21]  Ian Jones,et al.  Peer assessment without assessment criteria , 2014 .

[22]  Susan Bloxham,et al.  Mark my words: the role of assessment criteria in UK higher education grading practices , 2011 .

[23]  Nangkula Utaberta,et al.  Aligning Assessment with Learning Outcomes , 2012 .

[24]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[25]  Carol Boston.  Understanding Scoring Rubrics: A Guide for Teachers. , 2002 .

[26]  Ranald Macdonald,et al.  Changing Assessment in Higher Education: A Model in Support of Institution-Wide Improvement , 2009 .

[27]  Ernesto Panadero,et al.  Developing evaluative judgement , 2018 .

[28]  Joanna Bull,et al.  Assessing student learning in higher education , 1997 .

[29]  Harry Torrance.  Assessment as learning? How the use of explicit learning objectives, assessment criteria and feedback in post‐secondary education and training can come to dominate learning. 1 , 2007 .

[30]  M. Price,et al.  Let’s stop the pretence of consistent marking: exploring the multiple limitations of assessment criteria , 2016 .

[31]  A. Nitko.  Educational Assessment of Students , 1996 .

[32]  Kevin F. Collis,et al.  Evaluating the Quality of Learning: The SOLO Taxonomy , 1977 .

[33]  Linda A. Suskie.  Using Assessment Results to Inform Teaching Practice and Promote Lasting Learning , 2009 .

[34]  U. Grömping.  Dependence of Variable Importance in Random Forests on the Shape of the Regressor Space , 2009 .

[35]  Dannelle D. Stevens,et al.  Introduction to Rubrics: An Assessment Tool to Save Grading Time, Convey Effective Feedback, and Promote Student Learning , 2004 .

[36]  S. Messick.  Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[37]  Anders Jonsson,et al.  The use of scoring rubrics: Reliability, validity, and educational consequences , 2007 .

[38]  Lewis Elton.  Are UK degree standards going up, down or sideways? , 1998 .

[39]  David Boud.  Assessment could demonstrate learning gains, but what is required for it to do so? , 2018 .

[40]  D. Royce Sadler,et al.  Transforming Holistic Assessment and Grading into a Vehicle for Complex Learning , 2009 .

[41]  Harvey Woolf,et al.  Assessment criteria: reflections on current practices , 2004 .

[42]  L. Elton,et al.  Assessment in universities: a critical review of research , 2002 .

[43]  D. Boud,et al.  Developing Evaluative Judgement in Higher Education : Assessment for Knowing and Producing Quality Work , 2018 .

[44]  Darryl M. Hunter.  The Use of Holistic versus Analytic Scoring for Large-Scale Assessment of Writing , 1996, Canadian Journal of Program Evaluation.

[45]  Claudia Harsch,et al.  Comparing holistic and analytic scoring methods: issues of validity and reliability , 2013 .

[46]  H. Andrade,et al.  Student perspectives on rubric-referenced assessment , 2005 .

[47]  D. Sadler.  The futility of attempting to codify academic achievement standards , 2014 .

[48]  Anders Jonsson,et al.  The Use of Scoring Rubrics for Formative Assessment Purposes Revisited: A Review. , 2013 .

[49]  Ernesto Panadero,et al.  To rubric or not to rubric? The effects of self-assessment on self-regulation, performance and self-efficacy , 2014 .