Analysis of Discourse Features and Verification of Scoring Levels for Independent and Integrated Prototype Written Tasks for the New TOEFL®. TOEFL® Monograph Series. MS-30. ETS RM-05-13.

We assessed whether and how the discourse written for prototype integrated tasks (involving writing in response to print or audio source texts) field tested for the new TOEFL® differs from the discourse written for independent essays (i.e., the TOEFL essay). We selected 216 compositions written for 6 tasks by 36 examinees in a field test—representing Score Levels 3, 4, and 5 on the TOEFL essay—then coded the texts for lexical and syntactic complexity, grammatical accuracy, argument structure, orientations to evidence, and verbatim uses of source text. Analyses with nonparametric MANOVAs, following a 3-by-3 (task type by English proficiency level) within-subjects factorial design, showed that the discourse produced for the integrated writing tasks differed significantly at the lexical, syntactic, rhetorical, and pragmatic levels from the discourse produced in the independent essay on most of these variables. In certain analyses, these differences were also obtained across the 3 ESL proficiency levels.

[1]  Marion Crowhurst,et al.  AUDIENCE AND MODE OF DISCOURSE EFFECTS ON SYNTACTIC COMPLEXITY IN WRITING AT TWO GRADE LEVELS , 1979 .

[2]  Alister Cumming,et al.  A teacher-verification study of speaking and writing prototype tasks for a new TOEFL , 2004 .

[3]  Roz Ivanič,et al.  I am how I sound , 2001 .

[4]  Rebecca Moore Howard,et al.  Plagiarisms, Authorships, and the Academic Death Penalty , 1995, College English.

[5]  Michael H. Kutner.  Applied Linear Statistical Models , 1974 .

[6]  A. Pennycook.  Borrowing Others' Words: Text, Ownership, Memory, and Plagiarism , 1996 .

[7]  J. Copas,et al.  Applied Linear Statistical Models, 2nd Edition. , 1987 .

[8]  Ivar Bråten,et al.  Students' Strategic Use of Multiple Sources During Expository Text Reading: A Longitudinal Think-Aloud Study , 2003 .

[9]  P. Nation,et al.  Vocabulary size and use: Lexical richness in L2 written production , 1995 .

[10]  K. Hyland,et al.  Talking to the Academy , 1996 .

[11]  Ellen L. Barton,et al.  Evidentials, Argumentation, and Epistemological Stance , 1993, College English.

[12]  G. Thompson,et al.  Evaluation in the Reporting Verbs Used in Academic Papers. , 1991 .

[13]  The Influence of Generative Rhetoric on the Syntactic Maturity and Writing Effectiveness of College Freshmen. , 1979 .

[14]  Leslie Grant,et al.  Using Computer-Tagged Linguistic Features to Describe L2 Writing Differences , 2000 .

[15]  Stuart S. Yeh.  Empowering Education: Teaching Argumentative Writing to Cultural Minority Middle-School Students , 1998, Research in the Teaching of English.

[16]  T. Mccann.  Student Argumentative Writing Knowledge and Ability at Three Grade Levels , 1989, Research in the Teaching of English.

[17]  Pierre Legendre,et al.  DISTANCE‐BASED REDUNDANCY ANALYSIS: TESTING MULTISPECIES RESPONSES IN MULTIFACTORIAL ECOLOGICAL EXPERIMENTS , 1999 .

[18]  Kathleen Bardovi-Harlig,et al.  Attainment of Syntactic and Morphological Accuracy by Advanced Language Learners , 1989, Studies in Second Language Acquisition.

[19]  Cary H. Grobe,et al.  Syntactic Maturity, Mechanics of Writing, and Teachers' Quality Ratings. , 1979 .

[20]  Ruth E. Knudson.  Analysis of Argumentative Writing at Two Grade Levels , 1992 .

[21]  Ron Scollon.  As a matter of fact: The changing ideology of authorship and responsibility in discourse , 1994 .

[22]  Avon Crismore,et al.  Metadiscourse in Persuasive Writing , 1993 .

[23]  Sandra Ishikawa,et al.  Objective measurement of low-proficiency EFL narrative writing , 1995 .

[24]  Alister Cumming,et al.  Decision Making While Rating ESL/EFL Writing Tasks: A Descriptive Framework. , 2002 .

[25]  Walter Kintsch,et al.  Toward a model of text comprehension and production. , 1978 .

[26]  D. Biber,et al.  Styles of stance in English: Lexical and grammatical marking of evidentiality and affect , 1989 .

[27]  Alister Cumming,et al.  Assessing L2 writing: Alternative constructs and ethical dilemmas , 2002 .

[28]  B. Tabachnick,et al.  Using Multivariate Statistics , 1983 .

[29]  M. Seaman,et al.  Writing in the Secondary Foreign Language Classroom: The Effects of Prompts and Tasks on Novice Learners of French , 2000 .

[30]  Liz Hamp-Lyons,et al.  WRITING: COMPOSITION, COMMUNITY, AND ASSESSMENT , 2000 .

[31]  Liz Hamp-Lyons.  Assessing Second Language Writing in Academic Contexts , 1991 .

[32]  Charles W. Stansfield,et al.  A long-term research agenda for the Test of Written English , 1988 .

[33]  Yong-Won Lee,et al.  Score Reliability as an Essential Prerequisite for Validating New Writing and Speaking Tasks for TOEFL. , 2002 .

[34]  J. Hayes,et al.  Fluency in Writing , 2001 .

[35]  Ulla Connor,et al.  Linguistic/Rhetorical Measures for International Persuasive Student Writing , 1990 .

[36]  Douglas Biber,et al.  Variation across speech and writing: Methodology , 1988 .

[37]  Marti J. Anderson,et al.  A new method for non-parametric multivariate analysis of variance in ecology , 2001 .

[38]  Shunji Inagaki,et al.  Second Language Development in Writing: Measures of Fluency, Accuracy, and Complexity , 1998 .

[39]  Robert S. Carlisle.  The Writing of Anglo and Hispanic Elementary School Students in Bilingual, Submersion, and Regular Programs , 1989, Studies in Second Language Acquisition.

[40]  Cheryl A. Engber.  The relationship of lexical proficiency to the quality of ESL compositions , 1995 .

[41]  Louis Legendre,et al.  The Importance of Being Digital , 1963 .

[42]  Liz Hamp-Lyons,et al.  TOEFL 2000 : writing : composition, community, and assessment , 1997 .

[43]  M. A. Britt,et al.  Improving Students' Ability to Identify and Use Source Information , 2002 .

[44]  V. Barnett,et al.  Applied Linear Statistical Models , 1975 .

[45]  Richard H. Haswell.  Error and Change in College Student Writing , 1988 .

[46]  J. Terra.  David Medalla in London , 1995 .

[47]  Jan Svartvik,et al.  A __ comprehensive grammar of the English language , 1988 .

[48]  Alister Cumming.  LEARNING TO WRITE IN A SECOND LANGUAGE: TWO DECADES OF RESEARCH , 2001 .

[49]  Douglas Biber,et al.  Dimensions of register variation: Contents , 1995 .

[50]  Vladimir A. Plungian,et al.  The place of evidentiality within the universal grammatical space , 2001 .

[51]  T. Homburg.  Holistic Evaluation of ESL Compositions: Can It Be Validated Objectively? , 1984 .

[52]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[53]  Glenn D. Deckert.  Perspectives on Plagiarism from ESL Students in Hong Kong. , 1993 .

[54]  Kyle Perkins,et al.  Using Objective Methods of Attained Writing Proficiency to Discriminate among Holistic Evaluations. , 1980 .

[55]  S. Wineburg.  Historical Problem Solving: A Study of the Cognitive Processes Used in the Evaluation of Documentary and Pictorial Evidence , 1991 .

[56]  Lawrence T. Frase,et al.  COMPUTER ANALYSIS OF THE TOEFL TEST OF WRITTEN ENGLISH , 1998 .

[57]  Charlene Polio,et al.  Measures of Linguistic Accuracy in Second Language Writing Research , 1997 .

[58]  John Read,et al.  Assessing Vocabulary by John Read , 2000 .

[59]  Diane Larsen-Freeman,et al.  An ESL Index of Development , 1978 .

[60]  Joanna G. Crammond,et al.  The Uses and Complexity of Argument Structures in Expert and Student Persuasive Writing , 1998 .

[61]  Barry M. Kroll.  How College Freshmen View Plagiarism , 1988 .

[62]  Liz Hamp-Lyons,et al.  Communicative Writing Profiles: An Investigation of the Transferability of a Multiple‐Trait Scoring Instrument Across ESL Writing Assessment Contexts * , 1991 .

[63]  D. Biber.  Spoken and Written Textual Dimensions in English: Resolving the Contradictory Findings , 1986 .

[64]  R. Abdi.  Interpersonal metadiscourse: an indicator of interaction and identity , 2002 .

[65]  Batia Laufer,et al.  The Development of L2 Lexis in the Expression of the Advanced Learner , 1991 .

[66]  Ling Shi,et al.  Textual Borrowing in Second-Language Writing , 2004 .

[67]  P. Lennon,et al.  Error: Some Problems of Definition, Identification, and Distinction , 1991 .

[68]  Jill Burstein,et al.  Automated Essay Scoring : A Cross-disciplinary Perspective , 2003 .

[69]  Brian H. McArdle,et al.  FITTING MULTIVARIATE MODELS TO COMMUNITY DATA: A COMMENT ON DISTANCE‐BASED REDUNDANCY ANALYSIS , 2001 .

[70]  Geoff Thompson,et al.  Voices in the Text: Discourse Perspectives on Language Reports , 1996 .

[71]  Eileen Kintsch,et al.  Macroprocesses and Microprocesses in the Development of Summarization Skill , 1990 .

[72]  Jeanne D. Day,et al.  Teaching Summarization Skills: Influences of Student Ability Level and Strategy Difficulty , 1986 .

[73]  Ann L. Brown,et al.  Macrorules for summarizing texts: the development of expertise , 1983 .

[74]  Jennifer Wiley,et al.  Constructing arguments from multiple sources: Tasks that promote understanding and not just memory for text. , 1999 .

[75]  D. Larsen-Freeman,et al.  THE CONSTRUCTION OF A SECOND LANGUAGE ACQUISITION INDEX OF DEVELOPMENT1 , 1977 .

[76]  S. Toulmin.  The uses of argument , 1960 .

[77]  Benjamin A. Rifkin,et al.  Error Gravity: A Critical Review of Research Design. , 1995 .

[78]  Alister Cumming,et al.  Reading and summarizing challenging texts in first and second languages , 1989 .

[79]  Peter N. Winograd.  Strategic difficulties in summarizing texts , 1984 .

[80]  Gaea Leinhardt,et al.  Writing from Primary Documents , 1998 .

[81]  T. Levine,et al.  Eta Squared, Partial Eta Squared, and Misreporting of Effect Size in Communication Research , 2002 .