The Direct Assessment of Writing Skill: A Measurement Review

......................................................................... ·· Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...... . 'JYpes of Direct Assessment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Task 'JYpes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Evaluation Methods............................................................... 3 Reliability of Direct Assessments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Factors Influencing Reliability Estimates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Reading Reliability Estimates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Score Reliability Estimates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Reliabilities of Analytic Subscales . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Summary of Reliability Evidence .................................................... 12 Validity of Direct Assessments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Concurrent Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Predictive Validity ................................................................ 14 Incremental Validity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Validity of Analytic Subscores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Construct Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Content Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Summary of Validity Evidence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Technological Developments ............................................................ 18 Summary and Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

[1]  Edys S. Quellmalz,et al.  EFFECTS OF DISCOURSE AND RESPONSE MODE ON THE MEASUREMENT OF WRITING COMPETENCE , 1982 .

[2]  Charles E. Werts,et al.  Using Longitudinal Data to Estimate Reliability in the Presence of Correlated Measurement Errors , 1980 .

[3]  Lawrence T. Frase,et al.  Computer Aids for Text Assessment and Writing Instruction. , 1981 .

[4]  L. R. Markham,et al.  Influences of Handwriting Quality on Teacher Evaluation of Written Work1 , 1976 .

[5]  John A. Valentine,et al.  The College Entrance Examination Board , 1961 .

[6]  Ann F. Coward.  A Comparison of two Methods of Grading English Compositions , 1952 .

[7]  William E. Coffman.  On the Reliability of Ratings of Essay Examinations in English. , 1971 .

[8]  The Reliability of an Essay Test in English , 1935, The School Review.

[9]  Lawrence T. Frase.  Ethics of imperfect measures , 1981, IEEE Transactions on Professional Communication.

[10]  Edith M. Huddleston.  Measurement of Writing Ability at the College-Entrance Level , 1954 .

[11]  Bertram C. Bruce,et al.  Three perspectives on writing , 1982 .

[12]  Leonard S. Cahen,et al.  Educational Testing Service , 1970 .

[13]  James A. Anderson,et al.  AN INVESTIGATION OF THE RELIABILITY OF FIVE PROCEDURES FOR GRADING ENGLISH THEMES. , 1967 .

[14]  Paul B. Diederich,et al.  Measuring Growth in English , 1974 .

[15]  Alden J. Moe.  Analyzing Text with Computers. , 1980 .

[16]  S. A. Akeju,et al.  THE RELIABILITY OF GENERAL CERTIFICATE OF EDUCATION EXAMINATION ENGLISH COMPOSITION PAPERS IN WEST AFRICA1 , 1972 .

[17]  Judith Dozier Hackman,et al.  How Well Do Freshmen Write? Implications for Placement and Pedagogy. , 1977 .

[18]  Martin Chodorow,et al.  The EPISTLE Text-Critiquing System , 1982, IBM Syst. J..

[19]  Robert F. Conry,et al.  The British Columbia Assessment of Written Expression: General Report. , 1980 .

[20]  Robert Bracewell,et al.  Cognitive processes in composing and comprehending discourse , 1982 .

[21]  William E. Coffman.  ON THE VALIDITY OF ESSAY TESTS OF ACHIEVEMENT1 , 1966 .

[22]  Pamela A. Moss,et al.  A Comparison of Procedures to Assess Written Language Skills at Grades 4, 7, and 10. , 1982 .

[23]  Simplex Structure in the Grading of Essay Tests1 , 1966 .

[24]  E. B. Page,et al.  The use of the computer in analyzing student essays , 1968 .

[25]  William B. Michael,et al.  The Comparative Validity of the California State University and Colleges English Placement Test (CSUC-EPT) in the Prediction of Fall Semester Grade Point Average and English Course Grades of First-Semester Entering Freshmen , 1978 .

[26]  Judith A. Powills.  Holistic Essay Scoring: An Application of the Model for the Evaluation of Writing Ability and the Measurement of Growth in Writing Ability Over Time. , 1979 .

[27]  Henry B. Slotnick.  TOWARD A THEORY OF COMPUTER ESSAY GRADING , 1972 .

[28]  Douglas S. Finlayson,et al.  THE RELIABILITY OF THE MARKING OF ESSAYS , 1951 .

[29]  Keith T. Checketts,et al.  The Validity of Awarding Credit By Examination in English Composition , 1974 .

[30]  Edith M. Huddleston.  MEASUREMENT OF WRITING ABILITY AT THE COLLEGE-ENTRANCE LEVEL: OBJECTIVE VS. SUBJECTIVE TESTING TECHNIQUES , 1952 .

[31]  Henry B. Slotnick,et al.  Essay Grading by Computer: A Laboratory Phenomenon? , 1971, English Journal.

[32]  William B. Michael,et al.  A Comparison of the Reliability and Validity of Ratings of Student Performance on Essay Examinations by Professors of English and by Professors in Other Disciplines , 1980 .

[33]  H. Gulliksen.  Theory of mental tests , 1952 .

[34]  H. Breland,et al.  A COMPARISON OF DIRECT AND INDIRECT ASSESSMENTS OF WRITING SKILL , 1979 .

[35]  P. S. Gingrich,et al.  The writer's workbench: Computer aids for text analysis , 1982 .

[36]  Miles Myers,et al.  A Procedure for Writing Assessment and Holistic Scoring. , 1982 .

[37]  Everett M. Shepherd.  The Effect of the Quality of Penmanship on Grades , 1929 .