Exploring the role of assessment criteria during teachers’ collaborative judgement processes of students’ portfolios

Student portfolios are increasingly used for assessing student competences in higher education, but results about the construct validity of portfolio assessment are mixed. A prerequisite for construct validity is that the portfolio assessment is based on relevant portfolio content. Assessment criteria, are often used to enhance this condition. This study aims to identify whether assessment criteria can improve content, argumentation and communication during teacher moderation while judging student portfolios. Six teachers scored 32 student portfolios in dyads with and without assessment criteria. Their judgement processes were qualitatively analysed. Results indicated that the quality of their judgement processes was low, since teachers based their judgements mainly on their own personal opinion and less on evidence found in the portfolio. Teachers barely paid attention to quality checks and easily agreed with each other. When teachers used assessment criteria, the quality of their judgements slightly improved. They based their judgements more on relevant evidence, used less personal experiences and more often checked the quality of their judgement processes. It is concluded that the quality of teacher portfolio judgement is low, and that the use of assessment criteria can enhance its quality.

[1]  Susy Macqueen,et al.  Validity , 1973, Just Algorithms.

[2]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[3]  S. Messick.  Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[4]  M.F. van der Schaaf,et al.  Developing and validating a design for teacher portfolio assessment , 2008 .

[5]  K. Wolf,et al.  Teaching Portfolios: Purposes and Possibilities. , 1998 .

[6]  Maryl Gearhart,et al.  Looking at Student Work for Teacher Learning, Teacher Community, and School Reform , 2003 .

[7]  Douglas B. Clark,et al.  Analytic Frameworks for Assessing Dialogic Argumentation in Online Learning Environments , 2007 .

[8]  E. E. Jones,et al.  From Acts To Dispositions The Attribution Process In Person Perception1 , 1965 .

[9]  S. Erduran,et al.  TAPping into argumentation: Developments in the application of Toulmin's Argument Pattern for studying science discourse , 2004 .

[10]  Jeroen Janssen,et al.  Automatic coding of dialogue acts in collaboration protocols , 2008, Int. J. Comput. Support. Collab. Learn..

[11]  D. Borsboom,et al.  The concept of validity. , 2004, Psychological review.

[12]  J. Elander.  Student Assessment from a Psychological Perspective , 2004 .

[13]  Pamela A. Moss,et al.  An Integrative Approach to Portfolio Evaluation for Teacher Licensure , 1998 .

[14]  J Noak,et al.  Performance rating. , 1999, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[15]  Jae-Won Moon.  Reflection in Learning and Professional Development: Theory and Practice , 2005 .

[16]  Robert L. Linn,et al.  Performance Assessment: Policy Promises and Technical Measurement Standards , 1994 .

[17]  James Elander,et al.  An application of judgment analysis to examination marking in psychology. , 2002, British journal of psychology.

[18]  Kelly S. Shapley,et al.  Developing a Valid and Reliable Portfolio Assessment in the Primary Grades: Building on Practical Experience , 1999 .

[19]  Wil Meeus,et al.  Portfolio in Higher Education: Time for a Clarificatory Framework , 2006 .

[20]  W C Borman,et al.  Consistency of rating accuracy and rating errors in the judgment of human performance. , 1977, Organizational behavior and human performance.

[21]  Mark D. Reckase.  Portfolio Assessment: A Theoretical Esthnate of Score Reliability , 2005 .

[22]  Michael T. Kane,et al.  An argument-based approach to validity. , 1992 .

[23]  Nico Verloop,et al.  Cognitive Representations in Raters' Assessment of Teacher Portfolios. , 2005 .

[24]  Karen Sheingold,et al.  Reasoning About Evidence in Portfolios: Cognitive Foundations for Valid and Reliable Assessment , 1998 .

[25]  M. Brekelmans,et al.  Assessing the interpersonal competence of beginning teachers: the quality of the judgement process , 2005 .

[26]  S. Messick.  Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[27]  Paul A. Kirschner,et al.  Determining the quality of Competence Assessment Programs: A self-evaluation procedure , 2007 .

[28]  N. Engels,et al.  Validity and reliability of portfolio assessment in pre‐service teacher education , 2009 .

[29]  Angelo S. DeNisi,et al.  A cognitive view of the performance appraisal process: A model and research propositions , 1984 .

[30]  Allan S. Cohen,et al.  Threats to the Valid Use of Assessments , 1996 .

[31]  Lee Andresen,et al.  Experience-based learning , 2020, Understanding Adult Education and Training.

[32]  P. Freebody,et al.  Assessment as Judgment-in-Context: Analysing how teachers evaluate students' writing 1 , 2007 .

[33]  Chris Rust,et al.  Improving Students' Learning by Developing their Understanding of Assessment Criteria and Processes , 2003 .

[34]  Douglas B. Clark,et al.  Personally‐Seeded Discussions to Scaffold Online Argumentation , 2007 .