Preparing for the Speaking Tasks of the TOEFL iBT® Test: An Investigation of the Journeys of Chinese Test Takers

Language test preparation has often been studied within the consequential validity framework in relation to ethics, equity, fairness, and washback of assessment. The use of independent and integrated speaking tasks in the TOEFL iBT® test represents a significant development and innovation in assessing speaking ability in academic contexts. Integrated tasks that involve synthesizing and summarizing information presented in reading and listening materials have the potential to generate new test preparation strategies. This study investigated the experiences of over 1,500 Chinese test takers and 23 teachers who were preparing for the TOEFL iBT speaking tasks. It examined the frequency of use of a number of different test preparation activities and materials, reasons, and expectations for taking preparation courses and the features of preparation courses. In addition, we examined the usefulness of test preparation from two perspectives: students' and teachers' perceptions as well as the relationship between test preparation and performance. Data were collected via questionnaires, focus group discussions, interviews with test takers and teachers, and classroom observations. The data showed that (a) test preparation was a hugely complex, multiple-components construct, and teaching and learning test-taking strategies compose the most prominent feature of intensive preparation courses; (b) there were significant age-related differences in students' preparation activities and focuses, although with small effect sizes; (c) there was a high agreement between teachers and students in their views on the usefulness of test preparation activities; and (d) there existed only a weak relationship between test preparation and performance. The only significant predictor of students' test performance was the frequency of their use of the TOEFL Practice Online TPO® practice tests. The findings of the study can enhance our understanding of the pedagogical practices that characterize test preparation programs and contribute to the ongoing validity argument for the TOEFL iBT Speaking test. The implications of the findings for test publishers, test takers, teachers, and test preparation schools are discussed with reference to the instructional, learning, and affective aspects of the multifaceted construct of test preparation.

[1]  Richard K. Rein.  Educational Testing Service. The Examiner Examined. , 1974 .

[2]  Walter Kintsch,et al.  Summarizing Stories After Reading and Listening. , 1977 .

[3]  THE EFFECTS OF SPECIAL PREPARATION ON SAT‐VERBAL SCORES , 1979 .

[4]  S. Messick.  Test validity and the ethics of assessment. , 1980 .

[5]  Anne Anastasi,et al.  Coaching, test sophistication, and developed abilities. , 1981 .

[6]  Samuel Messick,et al.  Issues of effectiveness and equity in the coaching controversy: Implications for educational and testing practice , 1982 .

[7]  Donald E. Powers.  EFFECTS OF COACHING ON GRE APTITUDE TEST SCORES , 1983 .

[8]  Donald E. Powers.  Effects of Test Preparation on the Validity of a of a Graduate Admissions Test , 1985 .

[9]  PATTERNS OF TEST TAKING AND SCORE CHANGE FOR EXAMINEES WHO REPEAT THE TEST OF ENGLISH AS A FOREIGN LANGUAGE , 1987 .

[10]  Who Benefits Most From Preparing for a "Coachable" Admissions Test? , 1987 .

[11]  S. Messick.  Meaning and Values in Test Validation: The Science and Ethics of Assessment , 1989 .

[12]  Robert Keith Johnson,et al.  The Second Language Curriculum: Index , 1989 .

[13]  R. Oxford.  Use of language learning strategies: A synthesis of studies with implications for strategy training , 1989 .

[14]  W. A. Mehrens,et al.  Methods for Improving Standardized Test Scores: Fruitful, Fruitless, or Fraudulent? , 1989 .

[15]  R. Linn.  Admissions Testing: Recommended Uses, Validity, Differential Prediction, and Coaching. , 1990 .

[16]  Elazar J. Pedhazur,et al.  Measurement, Design, and Analysis: An Integrated Approach , 1994 .

[17]  W. James Popham,et al.  Appropriateness of Teachers' Test-Preparation Practices , 1991 .

[18]  M. Smith.  Meanings of Test Preparation , 1991 .

[19]  Elazar J. Pedhazur,et al.  Measurement, design, and analysis: An integrated approach, Student ed. , 1991 .

[20]  J. Charles Alderson,et al.  Does Washback Exist , 1993 .

[21]  J. Charles Alderson,et al.  Examining washback: the Sri Lankan Impact Study , 1993 .

[22]  A. Lehrer.  Understanding classroom lectures , 1994 .

[23]  Rebecca L. Oxford,et al.  A Closer Look at Learning Strategies, L2 Proficiency, and Gender , 1995 .

[24]  Bernard Spolsky,et al.  An Investigation into the Comparability of Two Tests of English as a Foreign Language , 1995 .

[25]  Michael Rost,et al.  Academic Listening: On-line summaries as representations of lecture understanding , 1995 .

[26]  Kathleen M. Bailey,et al.  Working for washback: a review of the washback concept in language testing , 1996 .

[27]  J. Charles Alderson,et al.  TOEFL preparation courses: a study of washback , 1996 .

[28]  Lyle F. Bachman,et al.  Language testing in practice : designing and developing useful language tests , 1996 .

[29]  Dianne Wall,et al.  Introducing new tests into traditional systems: insights from general education and from innovation theory , 1996 .

[30]  Samuel Messick.  Validity and washback in language testing , 1996 .

[31]  Paul Wadden,et al.  The Toefl and Its Imitators: Analyzing the Toefl and Evaluating Toefl-Prep Texts , 1997 .

[32]  L. Hamp-Lyons.  Washback, impact and validity: ethical concerns , 1997 .

[33]  College Students’ Beliefs about Exam Preparation , 1997 .

[34]  Dan Douglas,et al.  Testing speaking ability in academic contexts : theoretical considerations , 1997 .

[35]  P. Rea-Dickins.  So, why do we need relationships with stakeholders in language testing? A view from the UK , 1997 .

[36]  L. Hamp-Lyons.  Ethical Test Preparation Practice: The Case of the TOEFL , 1998 .

[37]  James Dh Brown.  Does IELTS preparation work?: An application of the context-adaptive model of language program evaluation , 1998 .

[38]  Comments on Liz Hamp-Lyons'"Ethical Test Preparation Practice: The Case of TOEFL. Polemic Gone Astray: A Corrective to Recent Criticism of TOEFL Preparation [and] the Author Responds. , 1999 .

[39]  Donald A. Rock,et al.  Effects of coaching on SAT I: Reasoning Test scores. , 1999 .

[40]  Mark Bray,et al.  The Shadow Education System: Private Tutoring and Its Implications for Planners , 1999 .

[41]  Tom Lumley,et al.  Conflicting perspectives on the role of test preparation in relation to learning , 2000 .

[42]  N. Postlethwaite.  Bray, Mark. 1999. The Shadow Education System: Private Tutoring and its Implications or Planners. Fundamentals of Educational Planning No. 61. Paris: UNESCO International Institute for Educational Planning. , 2000 .

[43]  A Framework for Evaluating the Validity of Test Preparation Practices. , 2000 .

[44]  N. Cole,et al.  The New Faces of Fairness , 2001 .

[45]  J. Stevens,et al.  Applied multivariate statistics for the social sciences, 4th ed. , 2002 .

[46]  Rae Everett,et al.  A critical analysis of selected IELTS preparation materials , 2003 .

[47]  The shape of things to come : will it be the normal distribution , 2004 .

[48]  J. Read,et al.  IELTS Test Preparation in New Zealand: Preparing Students for the IELTS Academic Module , 2004 .

[49]  The IELTS Impact Study: Investigating Washback on Teaching Materials , 2004 .

[50]  Donald E. Powers.  Coaching for the SAT: A Summary of the Summaries and an Update , 2005 .

[51]  A. Green.  EAP study recommendations and score gains on the IELTS Academic Writing test , 2005 .

[52]  T. Haladyna,et al.  Construct-Irrelevant Variance in High-Stakes Testing. , 2005 .

[53]  P. Rea-Dickins,et al.  Program Evaluation in Language Education , 2005 .

[54]  Linda Crocker,et al.  Teaching for the Test: Validity, Fairness, and Moral Action , 2005 .

[55]  M. Spratt.  Washback and the classroom: the implications for teaching and learning of studies of washback from exams , 2005 .

[56]  S. Ross.  L. Cheng, Y. Watanabe, and A. Curtis (eds): Washback in Language Testing: Research Contexts and Methods. Lawrence Erlbaum and Associates, 2004 , 2005 .

[57]  Steve Issitt,et al.  Improving scores on the IELTS speaking test , 2006 .

[58]  D. Wall,et al.  The Impact of Changes in the TOEFL Examination on Teaching and Learning in Central and Eastern Europe: Phase 1, The Baseline Study. TOEFL® Monograph Series. MS-34. ETS RR-06-18. , 2006 .

[59]  A. Green.  Washback to the learner: Learner and teacher perspectives on IELTS preparation course expectations and outcomes , 2006 .

[60]  P. Rea-Dickins,et al.  Washback from language tests on teaching, learning and policy: evidence from diverse settings , 2007 .

[61]  Anthony Green,et al.  Washback to learning outcomes: a comparative study of IELTS preparation and university pre‐sessional language courses , 2007 .

[62]  Yanren Ding,et al.  Text memorization and imitation: The practices of successful Chinese learners of English , 2007 .

[63]  D. Wall,et al.  THE IMPACT OF CHANGES IN THE TOEFL EXAMINATION ON TEACHING AND LEARNING IN CENTRAL AND EASTERN EUROPE: PHASE 2, COPING WITH CHANGE , 2008 .

[64]  The significance of sociolinguistic backgrounds of teachers of IELTS Test preparation courses in selected Malaysian institutions , 2008 .

[65]  S. Ross.  Language testing in Asia: Evolution, innovation, and policy challenges , 2008 .

[66]  Guoxing Yu,et al.  Reading to summarize in English and Chinese: a tale of two languages? , 2008 .

[67]  Peter Mickan,et al.  An ethnographic study of classroom instruction in an IELTS preparation program , 2008 .

[68]  Henry Latham,et al.  On The Action Of Examinations: Considered As A Means Of Selection , 2008 .

[69]  Gan Zhengdong,et al.  IELTS Preparation Course and Student IELTS Performance A Case Study in Hong Kong , 2009 .

[70]  Guoxing Yu,et al.  The Shifting Sands in the Effects of Source Text Summarizability on Summary Writing , 2009 .

[71]  David D. Qian.  From TOEFL pBT to TOEFL iBT : recent trends, research landscape, and Chinese learners , 2010 .

[72]  D. Wall,et al.  The Impact of Changes in the TOEFL® Exam on Teaching in a Sample of Countries in Europe: Phase 3, The Role of the Coursebook Phase 4, Describing Change , 2011 .

[73]  Richard Badger,et al.  To what extent is communicative language teaching a feature of IELTS classes in China , 2012 .

[74]  Kellie Frost,et al.  Investigating the validity of an integrated listening-speaking task: A discourse-based analysis of test takers’ oral performances , 2012 .

[75]  Donald E. Powers.  Understanding the Impact of Special Preparation for Admissions Tests , 2012 .

[76]  Danling Fu,et al.  Tests of English Language as significant thresholds for college-bound Chinese and the washback of test-preparation , 2012 .

[77]  Ets Spc,et al.  Understanding the Impact of Special Preparation for Admissions Tests , 2012 .

[78]  B. Tomlinson.  Materials development for language learning and teaching , 2012, Language Teaching.

[79]  From Integrative to Integrated Language Assessment: Are We There Yet? , 2013 .

[80]  M. Swain,et al.  Test-Takers' Strategic Behaviors in Independent and Integrated Speaking Tasks. , 2013 .

[81]  Guoxing Yu,et al.  The Use of Summarization Tasks: Some Lexical and Conceptual Analyses , 2013 .

[82]  Tim Farnsworth.  Effects of Targeted Test Preparation on Scores of Two Tests of Oral English as a Second Language , 2013 .

[83]  Qin Xie,et al.  Do test design and uses influence test preparation? Testing a model of washback with Structural Equation Modeling , 2013 .

[84]  Qin Xie.  Does Test Preparation Work? Implications for Score Validity , 2013 .

[85]  Rachel M. Adler,et al.  Do TOEFL iBT® Scores Reflect Improvement in English‐Language Proficiency? Extending the TOEFL iBT Validity Argument , 2014 .

[86]  O. Liu.  Investigating the Relationship Between Test Preparation and TOEFL iBT® Performance , 2014 .

[87]  Megan J. Montee,et al.  Stakeholders' Beliefs About the TOEFL iBT® Test as a Measure of Academic Language Ability , 2014 .

[88]  Investigating the Relationship between Test Preparation and "TOEFL iBT"® Performance. Research Report. ETS RR-14-15. , 2014 .

[89]  Guoxing Yu,et al.  English language assessment in China: policies, practices and impacts , 2014 .

[90]  Beata Beigman Klebanov,et al.  ETS RM – 1509 Examining Performance Differences on Tests of Academic English Proficiency Used for High-Stakes Versus Practice , 2015 .

[91]  Erik Voss,et al.  Utilizing Technology in Language Assessment , 2016 .

[92]  Jinsong Fan.  Assessing Chinese learners of English: language constructs, consequences and conundrums , 2017 .