Does calibration reduce variability in the assessment of accounting learning outcomes?

Reliable, consistent assessment process that produces comparable assessment grades between assessors and institutions is a core activity and an ongoing challenge with which universities have failed to come to terms. In this paper, we report results from an experiment that tests the impact of an intervention designed to reduce grader variability and develop a shared understanding of national threshold learning standards by a cohort of reviewers. The intervention involved consensus moderation of samples of accounting students’ work, with a focus on three research questions. First, what is the quantifiable difference in grader variability on the assessment of learning outcomes in ‘application skills’ and ‘judgement’? Second, does participation in the workshops lead to reduced disparity in the assessment of the students’ learning outcomes in ‘application skills’ and ‘judgement’? Third, does participation in the workshops lead to greater confidence by reviewers in their ability to assess students’ skills in application skills and judgement? Our findings suggest consensus moderation does reduce variability across graders and also builds grader confidence.

[1]  Peter Knight,et al.  The local practices of assessment , 2006 .

[2]  Cheryl Dhein,et al.  Outcomes assessment. , 2002, Journal of veterinary medical education.

[3]  M. Price,et al.  Assessment standards: the role of communities of practice and the scholarship of assessment , 2005 .

[4]  L. Elton,et al.  Assessment in universities: a critical review of research , 2002 .

[5]  Mantz Yorke,et al.  Assessment Learning and Employability , 2003 .

[6]  Rethinking university internationalisation: towards transformative change , 2011 .

[7]  Richard E. Baker,et al.  Summary of “outcomes assessment”: American Accounting Association Teaching and Curriculum Section outcomes assessment committee , 1994 .

[8]  Becky Francis,et al.  Gender, ‘bias’, assessment and feedback: analyzing the written assessment of undergraduate history essays , 2005 .

[9]  Phil Hancock,et al.  Learning and Teaching Academic Standards Statement for ACCOUNTING , 2010 .

[10]  External peer review of assessment: an effective approach to verifying standards? , 2015 .

[11]  David Baume,et al.  What is happening when we assess, and how can we use our understanding of this to improve assessment? , 2004 .

[12]  D. Royce Sadler,et al.  Specifying and Promulgating Achievement Standards , 1987 .

[13]  Susan Orr,et al.  Assessment moderation: constructing the marks and constructing the students , 2007 .

[14]  Susan Bloxham,et al.  Accountability in grading student work: securing academic standards in a twenty-first century quality assurance context , 2012 .

[15]  Mantz Yorke,et al.  Employability and Good Learning in Higher Education , 2003 .

[16]  John Biggs,et al.  The reflective institution: Assuring and enhancing the quality of teaching and learning , 2001 .

[17]  Nancy Allen,et al.  Assessment in Higher Education , 1992 .

[18]  P. Moss,et al.  Educational Standards, Assessment, and the Search for Consensus , 2001 .

[19]  D. Royce Sadler,et al.  Indeterminacy in the use of preset criteria for assessment and grading , 2009 .

[20]  Mantz Yorke.  Grading Student Achievement in Higher Education : Signals and Shortcomings , 2008 .

[21]  Erica Smith,et al.  Quality and Qualms in the Marking of University Assignments by Sessional Staff: An Exploratory Study , 2006 .

[22]  Chris Rust,et al.  The experience of introducing a common criteria assessment grid across an academic department , 1999 .

[23]  A. Wolf.  Competence-Based Assessment , 1995 .

[24]  Ian McKenna,et al.  Universities' contribution to the Bologna Process , 2005 .

[25]  James Elander,et al.  An application of judgment analysis to examination marking in psychology. , 2002, British journal of psychology.

[26]  Laura E. Rumbley,et al.  Trends in Global Higher Education: Tracking an Academic Revolution: A Report Prepared for the UNESCO 2009 World Conference on Higher Education , 2010 .

[27]  Chris Rust,et al.  Assessment standards: A potential role for Subject Networks , 2009 .

[28]  B. Visser,et al.  Emotional Intelligence: The Role of Accounting Education and Work Experience , 2011 .

[29]  Timothy L. Shaftel,et al.  Educational Assessment and the AACSB , 2007 .

[30]  Kerry Harman,et al.  Learning to assess in higher education: a collaborative exploration of the interplay of "formal" and "informal" learning in the academic workplace , 2010 .

[31]  D. Royce Sadler,et al.  Academic freedom, achievement standards and professional identity , 2011 .

[32]  Susan Bloxham,et al.  Marking and moderation in the UK: false assumptions and wasted resources , 2009 .

[33]  Phil Hancock,et al.  A Brave New World: Australian Learning Outcomes in Accounting Education , 2011 .

[34]  A. Abraham,et al.  Social moderation, assessment and assuring standards for accounting graduates , 2014 .

[35]  M. Price,et al.  External examining: fit for purpose? , 2015 .

[36]  D. Sadler.  The futility of attempting to codify academic achievement standards , 2014 .

[37]  D. Royce Sadler,et al.  Assessment in Education : Principles , Policy & Practice , 2012 .

[38]  Kathryn Ecclestone,et al.  'I know a 2:1 when I see it': Understanding criteria for degree classifications in franchised university programmes , 2001 .