Does an Argument-Based Approach to Validity Make a Difference?

Drawing on experience between 2000 and 2007 in developing a validity argument for the high-stakes Test of English as a Foreign Language™ (TOEFL®), this paper evaluates the differences between the argument-based approach to validity as presented byKane (2006)and that described in the 1999 AERA/APA/NCME Standards for Educational and Psychological Testing. Based on an analysis of four points of comparison—framing the intended score interpretation, outlining the essential research, structuring research results into a validity argument, and challenging the validity argument—we conclude that an argument-based approach to validity introduces some new and useful concepts and practices.

[1]  M. Kane.  Current Concerns in Validity Theory , 2001 .

[2]  Lyle F. Bachman.  Building and Supporting a Case for Test Use , 2005 .

[3]  D. Wall,et al.  The Impact of Changes in the TOEFL® Exam on Teaching in a Sample of Countries in Europe: Phase 3, The Role of the Coursebook Phase 4, Describing Change , 2011 .

[4]  Lyle F. Bachman.  Alternative Interpretations of Alternative Assessments: Some Validity Issues in Educational Performance , 2005 .

[5]  Lorrie A. Shepard,et al.  Chapter 9: Evaluating Test Validity , 1993 .

[6]  R. Almond,et al.  Focus Article: On the Structure of Educational Assessments , 2003 .

[7]  Cyril J. Weir,et al.  Language Testing and Validation , 2005 .

[8]  M. Swain,et al.  THEORETICAL BASES OF COMMUNICATIVE APPROACHES TO SECOND LANGUAGE TEACHING AND TESTING , 1980 .

[9]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.

[10]  C. Chapelle.  Construct definition and validity inquiry in SLA research , 1999 .

[11]  Michael T. Kane,et al.  Certification Testing as an Illustration of Argument-Based Validation , 2004 .

[12]  L. Shepard.  Chapter 9: Evaluating Test Validity , 1993 .

[13]  Denny Borsboom,et al.  The attack of the psychometricians , 2006, Psychometrika.

[14]  Allan S. Cohen,et al.  Validating Measures of Performance , 2005 .

[15]  Michael T. Kane,et al.  An argument-based approach to validity. , 1992 .

[16]  Michael T. Kane,et al.  Validating High-Stakes Testing Programs , 2005 .

[17]  S. Whitely.  Construct validity: Construct representation versus nomothetic span. , 1983 .

[18]  S. Messick.  The Interplay of Evidence and Consequences in the Validation of Performance Assessments , 1994 .

[19]  James Dean Brown,et al.  Examinee abilities and task difficulty in task-based second language performance assessment , 2002 .