Analyses of 10 raters' reasoning during think-aloud interviews provided evidence supporting a model of the fundamental processes involved in rating standards-based, nonprescriptive portfolios. Raters cycled iteratively among component processes and among subprocesses within them rather than following a linear sequence. This process model for portfolio rating provides a framework within which to conceptualize sound rater reasoning and to distinguish between acceptable variations and those that distort the meaning of scores. Score validity was threatened when raters omitted one of the major processes, evaluated portfolios without consideration of important criteria in the rating guide, or applied extraneous assessment criteria The 3 raters who exhibited these kinds of behaviors in their rating rationales generated ratings at a portfolio reading that were unacceptably inconsistent and unreliable. Furthermore, raters with identical patterns of problematic psychometric results were found to reason in substanti...
[1]
Joan L. Herman,et al.
Portfolio Research: A Slim Collection.
,
1994
.
[2]
Sandra Murphy,et al.
Portfolios and curriculum reform: Patterns in practice
,
1994
.
[3]
Edward W. Wolfe,et al.
The relationship between essay reading style and scoring proficiency in a psychometric scoring system
,
1997
.
[4]
S. Messick.
The Interplay of Evidence and Consequences in the Validation of Performance Assessments
,
1994
.
[5]
Anne Brown,et al.
The effect of rater variables in the development of an occupation-specific language performance test
,
1995
.
[6]
Joan L. Herman,et al.
Establishing Validity for Performance-Based Assessments: An Illustration for Collections of Student Writing.
,
1996
.
[7]
Pamela A. Moss,et al.
Can There Be Validity Without Reliability?
,
1994
.