Determining the quality of Competence Assessment Programs: A self-evaluation procedure

As assessment methods are changing, the way to determine their quality needs to be changed accordingly. This article argues for the use Competence Assessment Programs (CAPs), combinations of traditional tests and new assessment methods which involve both formative and summative assessments. To assist schools in evaluating their CAPs, a self-evaluation procedure was developed, based on 12 quality criteria for CAPs developed in earlier studies. A self-evaluation was chosen as it is increasingly used as an alternative to external evaluation. The CAP self-evaluation is carried out by a group of functionaries from the same school and comprises individual self-evaluations and a group interview. The CAP is rated on the 12 quality criteria and a piece of evidence is asked for to support these ratings. In this study, three functionaries from eight schools (N = 24) evaluated their CAP using the self-evaluation procedure. Results show that the group interview was very important as different perspectives on the CAP are assembled here into an overall picture of the CAP's quality. Schools seem to use mainly personal experiences to support their ratings and need to be supported in the process of carrying out a self-evaluation.

[1]  Filip Dochy,et al.  New Lines in Edumetrics: New Forms of Assessment Lead to New Assessment Criteria. , 2001 .

[2]  Michael T. Kane,et al.  Certification Testing as an Illustration of Argument-Based Validation , 2004 .

[3]  Robert Glaser,et al.  Assessment, Testing, and Instruction: Retrospect and Prospect. , 1994 .

[4]  Daniel T. Hickey,et al.  Balancing varied assessment functions to attain systemic validity: Three is the magic number , 2006 .

[5]  Mitchell D. Chester.  Multiple Measures and High‐Stakes Decisions: A Framework for Combining Measures , 2005 .

[6]  P. Knight.  The Value of a Programme-wide Approach to Assessment , 2000 .

[7]  S. Messick.  The Interplay of Evidence and Consequences in the Validation of Performance Assessments , 1994 .

[8]  David Gijbels,et al.  Assessment engineering: aligning assessment learning and instruction , 2004 .

[9]  David Baume,et al.  What is happening when we assess, and how can we use our understanding of this to improve assessment? , 2004 .

[10]  F. Dochy,et al.  Assessment as a tool for learning , 1997 .

[11]  Cees P. M. van der Vleuten,et al.  Assessing professional competence: from methods to programmes , 2005 .

[12]  Stephen B. Dunbar,et al.  Complex, Performance-Based Assessment: Expectations and Validation Criteria , 1991 .

[13]  P. Black,et al.  Assessment and Classroom Learning , 1998 .

[14]  Liesbeth Baartman,et al.  Estimating the quality of new modes of assessment , 2007 .

[15]  David Nevo.  0191-491 X(94) E0006-2 COMBINING INTERNAL AND EXTERNAL EVALUATION: A CASE FOR SCHOOL-BASED EVALUATION , 1994 .

[16]  Robert L. Linn,et al.  Validation of Performance-Based Assessments , 2000 .

[17]  Menucha Birenbaum,et al.  A learning Integrated Assessment System , 2006 .

[18]  Gijsbert Erkens,et al.  Teachers' Assessment of Students' Research Skills. , 2004 .

[19]  Pamela A. Moss,et al.  Can There Be Validity Without Reliability? , 1994 .

[20]  Ronald K. Hambleton,et al.  A Psychometric Perspective on Authentic Measurement , 1992 .

[21]  Cees P. M. van der Vleuten,et al.  Changing education, changing assessment, changing research? , 2004 .

[22]  Yves Benett,et al.  The Validity and Reliability of Assessments and Self‐assessments of Work‐based Learning , 1993 .

[23]  Linda S. Lotto.  Qualitative Data Analysis: A Sourcebook of New Methods , 1986 .

[24]  P. Kirschner,et al.  A five-dimensional framework for authentic assessment , 2004 .

[25]  Effie Maclellan.  Initial Knowledge States about Assessment: Novice Teachers' Conceptualisations. , 2004 .

[26]  David Nevo.  School Evaluation: Internal or External?. , 2001 .

[27]  P. Kirschner,et al.  Teachers ’ opinions on quality criteria for Competency Assessment Programmes , 2009 .

[28]  Linda R. Valli,et al.  New standards and assessments? Curriculum transformation in teacher education , 2002 .

[29]  M. Birenbaum.  Assessment 2000: Towards a Pluralistic Approach to Assessment , 1996 .

[30]  J. Charles Alderson,et al.  Does Washback Exist , 1993 .

[31]  S. Toulmin.  The uses of argument , 1960 .

[32]  Michael T. Kane,et al.  An argument-based approach to validity. , 1992 .

[33]  C. V. D. van der Vleuten,et al.  Changing education, changing assessment, changing research? , 2004, Medical education.

[34]  Paul A. Kirschner,et al.  The Wheel of Competency Assessment: Presenting Quality Criteria for Competency Assessment Programs. , 2006 .

[35]  M. Scriven.  Evaluation thesaurus, 4th ed. , 1991 .

[36]  G. McNamara,et al.  Internal review and self-evaluation — the chosen route to school improvement in ireland? , 2005 .

[37]  L. Kyriakides,et al.  School self-evaluation and school improvement: A critique of values and procedures , 2004 .

[38]  Keithia Wilson,et al.  Action Learning in Higher Education: an investigation of its potential to develop professional capability , 2004 .

[39]  Frans Meijers,et al.  Competencies as Building Blocks for Integrating Assessment with Instruction in Vocational Education: A case from The Netherlands , 2000 .

[40]  Shirley Simon,et al.  Learning to Teach Argumentation: Research and development in the science classroom , 2006 .

[41]  D. Kuhn.  THE SKILLS OF ARGUMENT , 2008, Education for Thinking.

[42]  Douglas B. Clark,et al.  Analyzing the quality of argumentation supported by personally-seeded discussions , 2005, CSCL.

[43]  J. Scheerens,et al.  Aspects of the Organizational and Political Context of School Evaluation in Four European Countries. , 1999 .