Using Global Complexity Measures to Assess Second Language Proficiency: Comparing CLIL and non-CLIL learners of Dutch and English in French-speaking Belgium

This study falls within the framework of an interdisciplinary project on Content and Language Integrated Learning (CLIL) in Frenchspeaking Belgium. One of the project’s aims is to compare the L2 language proficiency of CLIL and non-CLIL French-speaking learners of English and Dutch. In the present paper we focus on learners’ global proficiency and use of different types of metrics to assess syntactic and lexical complexity in the learners’ written productions. Using various computational tools, we extracted lexical and syntactic complexity scores for texts written by CLIL and non-CLIL pupils in their L2 (English or Dutch) and their L1 (French). These scores were then compared to investigate the potential influence of CLIL education on the pupils’ language proficiency as CLIL programs provide more target language input than non-CLIL programs. We therefore hypothesized that CLIL pupils would display a more native-like competence in the target language, i.e. a more native-like level of syntactic and lexical complexity in their writing. As for the influence of CLIL programs on the L1, we did not expect any difference between the two groups. Our results show that our first hypothesis is only partly confirmed as the effect of CLIL on L2 complexity varies according to the language: while the Dutch texts written by CLIL pupils turn out to be more complex for nearly all measures, this was only the case for half of the measures in the English 2 Amélie Bulon, Hendrikx, Meunier, Van Goethem texts. As initially expected for our second hypothesis, we found no influence of CLIL on the complexity of the pupils’ L1.

[1]  Xi Chen,et al.  Cross-Language Transfer of Morphological Awareness in Chinese-English Bilinguals. , 2011 .

[2]  Christiane Dalton-Puffer,et al.  The CLIL differential: Comparing the writing of CLIL and non-CLIL students in higher colleges of technology , 2010 .

[3]  T. McNamara,et al.  Assessed Levels of Second Language Speaking Proficiency: How Distinct? , 2007 .

[4]  David Marsh,et al.  CLIL/EMILE - the European dimension : actions, trends and foresight potential , 2002 .

[5]  K. W. Hunt.  Grammatical structures written at three grade levels , 1965 .

[6]  Jeanine Treffers-Daller.  Measuring lexical diversity among L2 learners of French: an exploration of the validity of D, MTLD and HD-D as measures of language ability , 2013 .

[7]  Philip M. McCarthy,et al.  MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment , 2010, Behavior research methods.

[8]  Ritualised routines and L2 acquisition: Acquisition strategies in an immersion program , 1991 .

[9]  Alex Housen,et al.  Evaluating short-term changes in L2 complexity development , 2015 .

[10]  Leslie Grant,et al.  Using Computer-Tagged Linguistic Features to Describe L2 Writing Differences , 2000 .

[11]  L. Ortega.  Syntactic Complexity Measures and Their Relationship to L2 Proficiency: A Research Synthesis of College-Level L2 Writing. , 2003 .

[12]  Danielle S. McNamara,et al.  Computational assessment of lexical differences in L1 and L2 writing , 2009 .

[14]  Alex Housen,et al.  Defining and operationalising L2 complexity , 2012 .

[15]  Fanny Meunier,et al.  Using global complexity measures to assess second language proficiency: Comparing CLIL and non-CLIL learners of English and Dutch in French-speaking Belgium , 2017 .

[16]  Laura K. Allen,et al.  Linguistic Microfeatures to Predict L2 Writing Proficiency: A Case Study in Automated Writing Evaluation. , 2014 .

[17]  J. Norris,et al.  Towards an Organic Approach to Investigating CAF in Instructed SLA: The Case of Complexity , 2009 .

[18]  Batia Laufer,et al.  The Relationship Between Passive and Active Vocabularies: Effects of LanguageLearning Context , 1998 .

[19]  Heidi Fox,et al.  Phrasal Cohesion and Statistical Machine Translation , 2002, EMNLP.

[20]  Danielle S. McNamara,et al.  Chapter 4. Validating lexical measures using human scores of lexical proficiency , 2013 .

[21]  Rie Koizumi,et al.  Relationships between text length and lexical diversity measures: Can we use short texts of less than 100 tokens? , 2012 .

[22]  Pauline Foster,et al.  Task Design and Second Language Performance: The Effect of Narrative Type on Learner Output , 2008 .

[23]  Folkert Kuiken,et al.  Complexity, Accuracy and Fluency: Definitions, Measurement and Research. , 2012 .

[24]  Danielle S. McNamara,et al.  Predicting the proficiency level of language learners using lexical indices , 2012 .

[25]  R. Hout,et al.  Lexical richness in the spontaneous speech of bilinguals , 2003 .

[26]  M. Juan-Garau,et al.  Writing Development Under CLIL Provision , 2015 .

[27]  Xiaofei Lu,et al.  Automatic analysis of syntactic complexity in second language writing , 2010 .

[28]  Fred Genesee,et al.  Second Language Learning in School Settings: Lessons from Immersion , 2014 .

[29]  Guoxing Yu,et al.  Lexical Diversity in Writing and Speaking Task Performances , 2010 .

[30]  Teresa Navés,et al.  CLIL in Catalonia: An Overview of Research Studies , 2010 .

[32]  Dana R. Ferris,et al.  Lexical and Syntactic Features of ESL Writing by Students at Different Levels of L2 Proficiency , 1994 .

[33]  Arthur C. Graesser,et al.  Coh-Metrix: Analysis of text on cohesion and language , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[34]  C. Pérez-Vidal,et al.  CLIL in Context: Profiling Language Abilities , 2015 .

[35]  Patrick Parslow,et al.  Back to Basics: How Measures of Lexical Diversity Can Help Discriminate between CEFR Levels , 2016 .

[36]  Elisabet Pladevall-Ballester.  Exploring primary school CLIL perceptions in Catalonia: students', teachers' and parents' opinions and expectations , 2015 .

[37]  A. E. Pierce,et al.  Language Acquisition and Syntactic Theory: A Comparative Analysis of French and English Child Grammars , 1992 .

[38]  Lourdes Ortega,et al.  Interlanguage complexity A construct in search of theoretical renewal , 2012 .

[39]  Thomas François,et al.  Les apports du traitement automatique des langues à la lisibilité du français langue étrangère , 2011 .

[40]  Nina Vyatkina,et al.  The Development of Second Language Writing Complexity in Groups and Individuals: A Longitudinal Learner Corpus Study , 2012 .