Another Look at Norris and Ortega (2000)

Since Norris and Ortega‟s (2000) seminal work on the effectiveness of second language instruction, there has been a proliferation of meta-analyses in the field of applied linguistics. Subsequent meta-analysts, however, have uncritically followed the methodological choices made by Norris and Ortega. This paper suggests a critical reevaluation of the methodological procedures underlying the Norris and Ortega (2000) meta-analysis. I reexamined their procedures, and reassessed the 49 unique samples they used in their meta-analysis. In doing so, I identified three key methodological limitations with the study, pertaining, respectively, to (a) the data collection procedure, (b) the coding system, and (c) the statistical analysis. I argue that the lack of data quality inherent in the primary studies, the oversimplified coding scheme, and the inappropriate use of effect size statistics combine to compromise the validity of the conclusions Norris and Ortega have drawn from their meta-analysis. I subsequently provide alternative procedures which may yield a more empirically sound research synthesis, recommending, for future meta-analysts, the „best evidence synthesis‟ approach where conclusions are drawn from combining quantitative and qualitative analyses.

[1]  Mikio Kubota.  The Garden Path Technique: Is It Really Effective?. , 1995 .

[2]  Catherine Doughty,et al.  Focus on Form in Classroom Second Language Acquisition. The Cambridge Applied Linguistics Series. , 1998 .

[3]  Talmy Givón,et al.  BENEFITS AND DRAWBACKS OF CONTROLLED LABORATORY STUDIES OF SECOND LANGUAGE ACQUISITION , 1997, Studies in Second Language Acquisition.

[4]  P. Robinson.  GENERALIZABILITY AND AUTOMATICITY OF SECOND LANGUAGE LEARNING UNDER IMPLICIT, INCIDENTAL, ENHANCED, AND INSTRUCTED CONDITIONS , 1997, Studies in Second Language Acquisition.

[5]  Jacquelyn Schachter,et al.  Second Language Acquisition and Its Relationship to Universal Grammar , 1988 .

[6]  M. Kubota.  Teachability of Conversational Implicature to Japanese EFL Learners. , 1995 .

[7]  R. Rosenthal.  The file drawer problem and tolerance for null results , 1979 .

[8]  Noriko Nagata,et al.  INPUT VS. OUTPUT PRACTICE IN EDUCATIONAL SOFTWARE FOR SECOND LANGUAGE ACQUISITION , 2005 .

[9]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[10]  Noriko Nagata,et al.  Intelligent Computer Feedback for Second Language Instruction , 1993 .

[11]  ZhaoHong Han,et al.  Fossilization in adult second language acquisition , 2004 .

[12]  Zoltán Dörnyei,et al.  Research methods in applied linguistics : quantitative,qualitative, and mixed methodologies , 2007 .

[13]  Sang-Ki Lee,et al.  VISUAL INPUT ENHANCEMENT AND GRAMMAR LEARNING: A Meta-Analytic Review , 2008, Studies in Second Language Acquisition.

[14]  R. Leow.  The Effects of Amount and Type of Exposure on Adult Learners' L2 Development in SLA , 1998 .

[15]  Bill VanPatten,et al.  Explicit Instruction and Input Processing , 1993, Studies in Second Language Acquisition.

[16]  Peter Master,et al.  Perspectives on Pedagogical Grammar: The effect of systematic instruction on learning the English article system , 1994 .

[17]  Peter Robinson,et al.  Consciousness, rules and instructed second language acquisition , 1996 .

[18]  Robert E. Slavin,et al.  Best-Evidence Synthesis: An Alternative to Meta-Analytic and Traditional Reviews , 1986 .

[19]  Nicole Tracy-Ventura,et al.  Investigating the empirical link between task-based interaction and acquisition: A meta-analysis , 2006 .

[20]  R. Leow.  Toward operationalizing the process of attention in SLA: Evidence for Tomlin and Villa's (1994) finegrained analysis of attention , 1998, Applied Psycholinguistics.

[21]  R. Leow.  Attention, Awareness, and Foreign Language Behavior , 1997 .

[22]  Claire J. Kramsch,et al.  Foreign language research in cross-cultural perspective , 1991 .

[23]  Alison Mackey,et al.  Conversational Interaction and Second Language Development: Recasts, Responses, and Red Herrings? , 1998 .

[24]  Alison Mackey,et al.  Interaction research in SLA : A meta-analysis and research synthesis , 2007 .

[25]  Gregg B. Jackson,et al.  Meta-Analysis: Cumulating Research Findings Across Studies , 1982 .

[26]  Nina Spada,et al.  4. The effectiveness of corrective feedback for the acquisition of L2 grammar: A meta-analysis of the research , 2006 .

[27]  Lester C. Loschky,et al.  Comprehensible Input and Second Language Acquisition , 1994, Studies in Second Language Acquisition.

[28]  Robert E. Slavin,et al.  Perspectives on Evidence-Based Research in Education—What Works? Issues in Synthesizing Educational Program Evaluations , 2008 .

[29]  Robert DeKeyser,et al.  Learning Second Language Grammar Rules , 1995, Studies in Second Language Acquisition.

[30]  Howard B. Lee,et al.  Foundations of Behavioral Research , 1973 .

[31]  Noriko Nagata,et al.  An Effective Application of Natural Language Processing in Second Language Instruction , 2013, CALICO Journal.

[32]  Harris Cooper,et al.  Scientific Guidelines for Conducting Integrative Research Reviews , 1982 .

[33]  Lawrence F. Bouton.  Can NNS Skill in Interpreting Implicature in American English Be Improved through Explicit Instruction?--A Pilot Study. , 1994 .

[34]  R. Dekeyser,et al.  The differential role of comprehension and production practice , 1996 .

[35]  Noriko Nagata,et al.  An Experimental Comparison of Deductive and Inductive Feedback Generated by a Simple Parser , 1997 .

[36]  M. Rafael Salaberry,et al.  The Role of Input and Output Practice in Second Language Acquisition. , 1997 .

[37]  Teresa Cadierno.  Formal Instruction from a Processing Perspective: An Investigation into the Spanish Past Tense , 1995 .

[38]  A. Agresti,et al.  Statistical Methods for the Social Sciences , 1979 .

[39]  Rick de Graaff,et al.  THE EXPERANTO EXPERIMENT , 1997, Studies in Second Language Acquisition.

[40]  Birgit Harley,et al.  Functional Grammar in French Immersion: A Classroom Experiment1 , 1989 .

[41]  Thomas H. Dinsmore.  2. Principles, parameters, and SLA: A retrospective meta-analytic investigation into adult L2 learners’ access to Universal Grammar , 2006 .

[42]  B. Green,et al.  Quantitative methods for literature reviews. , 1984, Annual review of psychology.

[43]  Elaine Day,et al.  Integrating formal and functional approaches to language teaching in French immersion : An experimental study , 1991 .

[44]  Fred R. Eckman,et al.  From Input to Output: Processing Instruction and Communicative Tasks , 2013 .

[45]  B. Vanpatten,et al.  Explanation versus Structured Input in Processing Instruction , 1996, Studies in Second Language Acquisition.

[46]  R. Dekeyser,et al.  BEYOND EXPLICIT RULE LEARNING , 1997, Studies in Second Language Acquisition.

[47]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[48]  Noriko Nagata.  The Effectiveness of Computer-Assisted Metalinguistic Instruction: A Case Study in Japanese , 1997 .

[49]  G. Glass.  Primary, Secondary, and Meta-Analysis of Research1 , 1976 .

[50]  Michael H. Long.  Focus on Form: A Design Feature in Language Teaching Methodology , 1991 .

[51]  Mikio Kubota,et al.  The Effects of Instruction Plus Feedback on Japanese University Students of EFL: A Pilot Study. , 1996 .

[52]  V. Scott.  An Empirical Study of Explicit and Implicit Teaching Strategies in French , 1989 .

[53]  Mikio Kubota.  The Role of Negative Feedback on the Acquisition of the English Dative Alternation by Japanese College Students of EFL. , 1994 .

[54]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[55]  R. Ellis.  Introduction: Investigating Form‐Focused Instruction , 2001 .

[56]  Michael H. Long.  Does Second Language Instruction Make a Difference? A Review of Research , 1983 .

[57]  Michael H. Long,et al.  The Role of Implicit Negative Feedback in SLA: Models and Recasts in Japanese and Spanish , 1998 .

[58]  M. Borenstein,et al.  Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments , 2006 .

[59]  Tadayoshi Kaya,et al.  5. Effects of L2 instruction on interlanguage pragmatic development: A meta-analysis , 2006 .

[60]  John Truscott,et al.  The effect of error correction on learners' ability to write accurately , 2007 .

[61]  J. Hulstijn,et al.  Implicit and incidental second language learning: experiments in the processing of natural and partly artificial input , 1989 .

[62]  L. Hedges.  Distribution Theory for Glass's Estimator of Effect size and Related Estimators , 1981 .

[63]  Rod Ellis,et al.  Communicating about Grammar: A Task-Based Approach , 1991 .

[64]  R. Lyster.  The Effect of Functional-Analytic Teaching on Aspects of French Immersion Students' Sociolinguistic Competence. , 1994 .

[65]  C. Doughty.  Second Language Instruction Does Make a Difference , 1991, Studies in Second Language Acquisition.

[66]  B. Vanpatten,et al.  Input Processing and Grammar Instruction: Theory and Research , 1998 .

[67]  Susanne Carroll,et al.  The role of feedback in adult second language acquisition: Error correction and morphological generalizations , 1992, Applied Psycholinguistics.

[68]  Lourdes Ortega,et al.  Synthesizing research on language learning and teaching , 2006 .

[69]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[70]  P. Lee,et al.  Publication bias in meta-analysis: its causes and consequences. , 2000, Journal of clinical epidemiology.

[71]  L. Hedges,et al.  The Handbook of Research Synthesis , 1995 .

[72]  M. Swain,et al.  Explicit and Implicit Negative Feedback , 1993, Studies in Second Language Acquisition.

[73]  Terence Odlin,et al.  Perspectives on Pedagogical Grammar: GRAMMAR, LEXICON, AND DISCOURSE , 1994 .