The association between SAT prompt characteristics, response features, and essay scores

Abstract This study investigated the relationship of prompt characteristics and response features with essay scores on the SAT Reasoning Test. A sample of essays was coded on a variety of features regarding their length and content. Analyses included descriptive statistics and computation of effect sizes, correlations between essay features and scores, and hierarchical linear modeling to explore variation across prompts. The results indicate that essay length is related to scores, but the correlation is not nearly as high as previous critics have claimed. After controlling for SAT Critical Reading and Writing multiple-choice scores, the essay features with the largest positive effect sizes included using a five-paragraph theme (FPT) and using academic evidence. The features with the largest negative effect sizes included using no evidence or support, and ending the essay mid-sentence. The relationship of essay length and performance varied significantly across prompts, and this variation was explained by the average SAT Critical Reading performance of examinees for the prompt.

[1]  Carol O. Sweedler-Brown.  The Effect of Training on the Appearance Bias of Holistic Essay Graders. , 1992 .

[2]  Donald E. Powers,et al.  Will They Think Less of My Handwritten Essay If Others Word Process Theirs? Effects on Essay Scores of Intermingling Handwritten and Word-Processed Essays. , 1992 .

[3]  Dennis Briggs,et al.  THE INFLUENCE OF HANDWRITING ON ASSESSMENT , 1970 .

[4]  Thomas E. Nunnally.  Breaking the Five-Paragraph-Theme Barrier. , 1991 .

[5]  Eli Hinkel,et al.  Second Language Writers' Text: Linguistic and Rhetorical Features , 2002 .

[6]  An Analysis of English Composition Test Essay Prompts for Differential Difficulty. College Board Report No. 92-4. , 1992 .

[7]  H. Breland.  A Study of Gender and Performance on Advanced Placement History Examinations. College Board Report No. 91-4. , 1991 .

[8]  L. Hamp-Lyons.  What is writing? What is “scholastic aptitude”? What are the consequences? SAT I Writing — a trip down memory lane , 2005 .

[9]  Barbara Kroll.  Second Language Writing. Research Insights for the Classroom. , 1990 .

[10]  Alister Cumming,et al.  Expertise in evaluating second language compositions , 1990 .

[11]  Chaitra M. Hardison,et al.  Use of Writing Samples on Standardized Tests: Susceptibility to Rule-Based Coaching and the Resulting Effects on Score Improvement , 2008 .

[12]  Pamela A. Moss,et al.  Can There Be Validity Without Reliability? , 1994 .

[13]  Hunter M. Breland,et al.  Remote Scoring of Essays. College Board Report No. 88-3. , 1988 .

[14]  Edward W. Wolfe,et al.  Learning To Rate Essays: A Study of Scorer Cognition. , 1994 .

[15]  B. Huot,et al.  The Literature of Direct Writing Assessment: Major Concerns and Prevailing Trends , 1990 .

[16]  Peggy O'Neill,et al.  Assessing Writing: A Critical Sourcebook , 2008 .

[17]  W. Popham.  Teaching to the Test , 2001 .

[18]  Cindy Moore,et al.  A Usable Past for Writing Assessment. , 2010 .

[19]  Jennifer L. Kobrin,et al.  Validity of the SAT for Predicting First-Year College Grade Point Average , 2008 .

[20]  B. Bridgeman,et al.  Choice Among Essay Topics: Impact on Performance and Validity , 1997 .

[21]  J. M. Ryan,et al.  The Critical Role of Anchor Paper Selection in Writing Assessment , 2009 .

[22]  Ling Shi,et al.  Native- and nonnative-speaking EFL teachers’ evaluation of Chinese students’ English writing , 2001 .

[23]  L. R. Markham,et al.  Influences of Handwriting Quality on Teacher Evaluation of Written Work1 , 1976 .

[24]  Brian F. Patterson,et al.  Validity of the SAT for Predicting FYGPA: 2007 SAT Validity Sample. Statistical Report No. 2009-1. , 2009 .

[25]  J. Kobrin,et al.  SAT Writing: An Overview of Research and Psychometrics to Date , 2007 .

[26]  Emily J. Shaw,et al.  Does Quantity Equal Quality?: The Relationship between Length of Response and Scores on the SAT Essay , 2007 .

[27]  Bonnie Albertson,et al.  Organization and Development Features of Grade 8 and Grade 10 Writers: A Descriptive Study of Delaware Student Testing Program (DSTP) Essays. , 2007 .

[28]  James Hoetker,et al.  The Effects of Systematic Variations in Essay Topics on the Writing Performance of College Freshmen. , 1989 .

[29]  Kimberly Wesley.  The Ill Effects of the Five Paragraph Theme , 2000 .

[30]  Elana Shohamy,et al.  The test-takers' choice: an investigation of the effect of topic on language-test performance , 1999 .

[31]  Edward L. Korn,et al.  Analysis of Health Surveys , 1999 .

[32]  Alister Cumming,et al.  Decision Making While Rating ESL/EFL Writing Tasks: A Descriptive Framework. , 2002 .

[33]  Dorothy Worden,et al.  Finding process in product: Prewriting and revision in timed essay responses , 2009 .

[34]  K. Barkaoui,et al.  Participants, Texts, and Processes in ESL/EFL Essay Tests: A Narrative Review of the Literature , 2007 .

[35]  Harvey S. Wiener,et al.  Writing Assessment: Issues and Strategies , 1987 .

[36]  Jo Morrison,et al.  Use of an aptitude test in University entrance: a validity study , 2010 .

[37]  D. Borsboom.  Educational Measurement (4th ed.) , 2009 .

[38]  J. Mazzeo,et al.  Sex-Related Performance Differences on Constructed-Response and Multiple-Choice Sections of Advanced Placement Examinations. College Board Report No. 92-7. , 1993 .

[39]  R. Linn.  Educational measurement, 3rd ed. , 1989 .

[40]  Liz Hamp-Lyons,et al.  Second Language Writing: Second language writing: assessment issues , 1990 .

[41]  James A. Penny.  Reading high stakes writing samples: My life as a reader , 2003 .

[42]  Jeff Connor-Linton.  Crosscultural comparison of writing standards: American ESL and Japanese EFL , 1995 .

[43]  Edward Tokar,et al.  THE EFFECT OF THE QUALITY OF PRECEDING RESPONSES ON THE GRADES ASSIGNED TO SUBSEQUENT RESPONSES TO AN ESSAY QUESTION , 1975 .

[44]  B. Tuck,et al.  THE INFLUENCE OF CONTEXT POSITION AND SCORING METHOD ON ESSAY SCORING , 1980 .

[45]  John A. Daly,et al.  Contrast Effects in Evaluating Essays. , 1982 .

[46]  Brian F. Patterson,et al.  Differential Validity and Prediction of the SAT , 2008 .

[47]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .