Reconsidering the Measurement of Proficiency in Pilot and Air Traffic Controller Radiotelephony Communication: From Construct Definition to Task Design

The high-stakes context of international radiotelephony (RT) communication, in which pilots and air traffic controllers (ATCO) use aviation English (AE) as a lingua franca, requires a robust testing policy that is clear and fair to all stakeholders. The International Civil Aviation Organization Language Proficiency Requirements have been criticized for their lack of fit with pilots' and ATCOs' real-life communicative needs, for both native and non-native speakers of English (Douglas, 2014; Kim, 2012). This dissertation investigates and specifies the proficiency construct (awareness, knowledge, skills, and attitudes) in pilot-ATCO intercultural RT, following Fulcher and Davidson's (2007) test development framework, and drawing on theoretical and empirical studies in the domains of Aviation English, English as a Lingua Franca, Intercultural awareness, and Interactional competence. It is guided by the following overarching research question: What are the communicative demands of pilots and ATCOs involved in intercultural RT communications; how can they be specified within a construct framework and operationalized as test tasks? In order to address this question, a multiphase mixed methods (MM) research design (Creswell, 2014) with three integrated phases is applied. In phase one, the intercultural RT context is explored (MM exploratory study). In phase two, models of language use relevant to this aviation workplace are proposed; and, key construct components are specified and validated by 128 aviation stakeholders (qualitative study). In the third/final phase, draft tasks that operationalize the identified RT construct are designed and pilot tested with AE testing experts (convergent parallel MM study). Integration of all findings allows for meta-inferences regarding the research question and underscores 1) the value of MM research designs in addressing such complex questions, and 2) the role of domain experts in: exploring a target testing context, specifying and validating a construct, and designing and pilot testing tasks to generate the evidence needed to make valid inferences about test-takers' specific purpose language ability. A broader view of professional communicative competence for intercultural RT suggests an encouraging perspective in a highly specialized occupational domain, has implications for aviation safety, and serves as an example of how non-linguistic competencies could be operationalized in contexts other than aviation.

[1]  D. Douglas.  Assessing the language of International Civil Aviation: issues of validity and impact , 2004, International Professional Communication Conference, 2004. IPCC 2004. Proceedings..

[2]  J. Charles Alderson.  The Politics of Aviation English Testing , 2011 .

[3]  Emotional Branding on Social Media: A Cross-Cultural Discourse Analysis of Global Brands on Twitter and Weibo , 2018 .

[4]  Glenn Fulcher,et al.  Test architecture, test retrofit , 2009 .

[5]  神保 尚武.  言語運用能力(Communicative Competence) , 1987 .

[6]  T. McNamara.  Managing learning: Authority and language assessment , 2011, Language Teaching.

[7]  C. Candlin,et al.  Mixed-methods research in language teaching and learning: Opportunities, issues and challenges , 2014, Language Teaching.

[8]  Zhu Hua.  Identifying Research Paradigms , 2015 .

[9]  J. C. Alderson,et al.  The challenge of diagnostic testing:do we know what we are measuring , 2007 .

[10]  Lyle F. Bachman.  3. What is the Construct? The Dialectic of Abilities and Contexts in Defining Constructs in Language Assessment , 2007 .

[11]  John W. Creswell,et al.  Designing and Conducting Mixed Methods Research , 2006 .

[12]  S. O'Hagan,et al.  Extending the scope of speaking assessment criteria in a specific-purpose language test: Operationalizing a health professional perspective , 2016 .

[13]  Elana Shohamy,et al.  ELF and critical language testing , 2017 .

[14]  Dan Douglas,et al.  Assessing the communication skills of veterinary students: Whose criteria? , 2000 .

[15]  L. Taylor.  Washback and impact , 2005 .

[16]  L. Giddings.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, 2d ed , 2005 .

[17]  L. Harding.  Communicative Language Testing: Current Issues and Future Research , 2014 .

[18]  Dianne Wall,et al.  The Impact of High-Stakes Examinations on Classroom Teaching: A case study using insights from testing and innovation theory , 2006 .

[19]  C. Teddlie,et al.  Putting the Human Back in ‘‘Human Research Methodology’’: The Researcher in Mixed Methods Research , 2010 .

[20]  Michael D Fetters,et al.  Achieving integration in mixed methods designs-principles and practices. , 2013, Health services research.

[21]  Lyle F. Bachman,et al.  语言测试实践 = Language testing in practice , 1998 .

[22]  Brian K. Lynch,et al.  Testcraft: A Teacher`s Guide to Writing and Using Language Test Specifications , 2001 .

[23]  Alan Davies,et al.  ASSESSING ENGLISH AS A LINGUA FRANCA , 2006, Annual Review of Applied Linguistics.

[24]  J. Laborda,et al.  Building a Validity Argument for the Test of English as a Foreign Language , 2009 .

[25]  Manfred Max Bergman,et al.  7 Quality of Inferences in Mixed Methods Research: Calling for an Integrative Framework , 2008 .

[26]  C. Elder,et al.  Language Testing and English as an International Language Constraints and Contributions , 2008 .

[27]  Hyejeong Kim.  Exploring the construct of radiotelephony communication: A critique of the ICAO English testing policy from the perspective of Korean aviation experts , 2012 .

[28]  A. Davies.  ASSESSING WORLD ENGLISHES , 2009, Annual Review of Applied Linguistics.

[29]  W. Baker.  From cultural awareness to intercultural awareness: culture in ELT , 2012 .

[30]  Alison Noble,et al.  Communication for Maritime Purposes: Some Exploratory Results of A Survey-Based Study on Intercultural and Linguistic Features , 2011 .

[31]  C. Elder,et al.  The hunt for “indigenous criteria” in assessing communication in the physiotherapy workplace , 2016 .

[32]  R L Helmreich,et al.  Anatomy of a system accident: the crash of Avianca Flight 052. , 1994, The International journal of aviation psychology.

[33]  B. Seidlhofer.  Accommodation and the idiom principle in English as a Lingua Franca , 2009 .

[34]  W. Baker.  English as a lingua franca and intercultural communication , 2017 .

[35]  Nicole Barrette-Sabourin.  ICAO Language Proficiency Requirements , 2008 .

[36]  J. Jenkins,et al.  Review of developments in research into English as a lingua franca , 2011, Language Teaching.

[37]  Hyejeong Kim.  UNDERSTANDING AVIATION ENGLISH AS A LINGUA FRANCA PERCEPTIONS OF KOREAN AVIATION PERSONNEL , 2016 .

[38]  M. Byram.  Teaching and Assessing Intercultural Communicative Competence , 1997 .

[39]  T. McNamara.  Language assessment in foreign language education: The struggle over constructs , 2007 .

[40]  Markus Bieswanger.  Aviation English : two distinct specialized registers , 2016 .

[41]  Johan M. Sanne,et al.  Creating Safety in Air Traffic Control , 2001 .

[42]  I. Kecskés,et al.  Zufferey, Sandrine: Acquiring pragmatics. Social and cognitive perspectives , 2017 .

[43]  Descriptors Educational,et al.  of Educational Measurement , 1988 .

[44]  J. House,et al.  The pragmatics of ELF , 2017 .

[45]  Suresh Canagarajah,et al.  Changing Communicative Needs, Revised Assessment Objectives: Testing English as an International Language , 2006 .

[46]  Julie Pallant,et al.  SPSS survival manual : a step by step guide to data analysis using SPSS for Windows , 2001, Behaviour Change.

[47]  G. Kasper,et al.  Speaking in turns and sequences: Interactional competence as a target construct in testing speaking , 2018, Language Testing.

[48]  J. Morse.  Critical Analysis of Strategies for Determining Rigor in Qualitative Inquiry , 2015, Qualitative health research.

[49]  B. Seidlhofer.  Closing A Conceptual Gap: The Case For A Description Of English As A Lingua Franca , 2001 .

[50]  Penelope Brown,et al.  Politeness: Some Universals in Language Usage , 1989 .

[51]  Marko Dragojevic,et al.  Communication Accommodation Theory , 2015 .

[52]  M. Swain,et al.  THEORETICAL BASES OF COMMUNICATIVE APPROACHES TO SECOND LANGUAGE TEACHING AND TESTING , 1980 .

[53]  G. Hofstede,et al.  Cultures and Organizations: Software of the Mind , 1991 .

[54]  Robert L. Helmreich,et al.  Human Factors on the Flight Deck , 1996 .

[55]  Evelina D. Galaczi,et al.  Rating scale development: a multistage exploratory sequential design , 2016 .

[56]  Alireza Hazrati,et al.  Intercultural Communication and Discourse Analysis: The Case of Aviation English , 2015 .

[57]  Monika Richter,et al.  Cognition In The Wild , 2016 .

[58]  Edwin Hutchins,et al.  How a Cockpit Remembers Its Speeds , 1995, Cogn. Sci..

[59]  Lyle F. Bachman.  语言测试要略 = Fundamental considerations in language testing , 1990 .

[60]  W. Paul Vogt,et al.  Quantitative research methods for professionals , 2006 .

[61]  W. Baker.  Culture and language in intercultural communication, English as a lingua franca and English language teaching: points of convergence and conflict , 2016 .

[62]  S. Youn.  Validating task-based assessment of L2 pragmatics in interaction using mixed methods , 2013 .

[63]  Csr Young,et al.  How to Do Things With Words , 2009 .

[64]  Jonathan Culpeper,et al.  Towards an anatomy of impoliteness. , 1996 .

[65]  Barry O'Sullivan,et al.  Using observation checklists to validate speaking-test tasks , 2002 .

[66]  Hyejeong Kim.  What constitutes professional communication in aviation: Is language proficiency enough for testing purposes? , 2018, Language Testing.

[67]  C. Alptekin.  Towards intercultural communicative competence in ELT , 2002 .

[68]  Min-Sun Kim,et al.  Culture‐Based Conversational Constraints Theory , 2017 .

[69]  S. Messick.  The Interplay of Evidence and Consequences in the Validation of Performance Assessments , 1994 .

[70]  Niina Hynninen,et al.  The Common European Framework of Reference from the perspective of English as a lingua franca: what we can learn from a focus on language regulation , 2014 .

[71]  Gibson Ferguson.  The Routledge handbook of English as a lingua franca , 2018 .

[72]  Dan Douglas,et al.  Assessing Languages for Specific Purposes , 1999 .

[73]  Gert Jan Hofstede,et al.  Cultures and Organizations: Software of the Mind, 3rd ed. , 2010 .

[74]  Don Snow.  Intercultural Communication in English Courses in Asia: What Should We Teach About? , 2018 .

[75]  Fred Davidson,et al.  Language Testing and Assessment: An Advanced Resource Book , 2007 .

[76]  D. Deardorff.  Identification and Assessment of Intercultural Competence as a Student Outcome of Internationalization , 2006 .

[77]  Noriko Iwashita,et al.  Interactional competence: Genie out of the bottle , 2018, Language Testing.

[78]  A. Tajima,et al.  Fatal miscommunication: English in aviation safety , 2004 .

[79]  Donal Carbaugh.  Cultural Discourse Analysis: Communication Practices and Intercultural Encounters , 2007 .

[80]  J. Pallant.  SPSS survival manual : a step-by-step guide to data analysis using SPSS version 15 , 2007 .

[81]  J. Jenkins.  Current Perspectives on Teaching World Englishes and English as a Lingua Franca , 2006 .

[82]  Fred Dervin,et al.  Introduction - English as a lingua franca and interculturality : beyond orthodoxies. , 2016 .

[83]  B. Seidlhofer.  English as a lingua franca , 2005 .

[84]  T. McNamara.  English as a lingua franca: the challenge for language testing , 2012 .

[85]  Trinidad Lewis.  Intercultural Communication: A Discourse Approach , 1996 .

[86]  From Diagnosis Toward Academic Support: Developing a Disciplinary, ESP-Based Writing Task and Rubric to Identify the Needs of Entering Undergraduate Engineering Students , 2017 .

[87]  Donal Carbaugh.  Comments on “culture”; in communication inquiry , 1988 .

[88]  M. Maggetti.  Mixed-Methods Designs , 2020, Handbuch Methoden der Politikwissenschaft.

[89]  L. Taylor.  The changing landscape of English: implications for language assessment , 2006 .

[91]  Heather Murray,et al.  The Globalization of English and the English Language Classroom , 2006 .

[92]  Liying Cheng,et al.  Changing Language Teaching through Language Testing: A Washback Study , 2006 .

[93]  Liying Cheng,et al.  Assessment in the Language Classroom , 2017 .

[94]  Dan Douglas.  Nobody Seems to Speak English Here Today: Enhancing Assessment and Training in Aviation English , 2014 .

[95]  Brian Paltridge,et al.  The handbook of English for specific purposes , 2012 .

[96]  Michael Byram,et al.  Developing the Intercultural Dimension in Language Teaching: a practical introduction for teachers , 2002 .

[97]  Michael Rube Redfield,et al.  Language Test Construction and Evaluation , 1997 .

[98]  David Nash.  Aviation English: A Lingua Franca for Pilots and Air Traffic Controllers , 2017 .

[99]  Johnny Saldaña,et al.  The Coding Manual for Qualitative Researchers , 2009 .

[100]  Zhu Qiyu,et al.  A Guide to Doing Statistics in Second Language Research Using SPSS , 2011 .

[101]  Liying Cheng,et al.  Review of doctoral research in language assessment in Canada (2006–2011) , 2013, Language Teaching.

[102]  Elizabeth Mathews,et al.  Language analysis : there's more to aviation English than radio conversations between pilots and air traffic controllers , 2014 .

[103]  C. Juchem-Grundmann,et al.  Introduction , 2019 .

[104]  Elinor Ochs,et al.  Co-Construction: An Introduction , 1995 .

[105]  Jennifer Jenkins,et al.  The Phonology of English as an International Language: New Models, New Norms, New Goals , 2000 .

[106]  A. Onwuegbuzie,et al.  The Validity Issue in Mixed Research , 2006 .

[107]  W. Baker.  Culture and complexity through English as a lingua franca: rethinking competences and pedagogy in ELT , 2015 .

[108]  Zhu Hua,et al.  Research methods in intercultural communication: a practical guide , 2015 .

[109]  David E. Falkner.  Other , 2003, The Patrick Moore Practical Astronomy Series.

[110]  L. Harding,et al.  Language assessment : The challenge of ELF , 2017 .

[111]  Susy Macqueen,et al.  Validity , 1973, Just Algorithms.

[112]  L. Harding.  Adaptability and ELF communication:the next steps for communicative language testing? , 2015 .

[113]  Eveline Wyss-Bühlmann.  Variation and Co-operative Communication Strategies in Air Traffic Control English , 2004 .

[114]  Ashleigh Merritt.  CROSS-CULTURAL FACTORS IN AVIATION SAFETY , 2004 .

[115]  Carol Lynn Moder,et al.  Planes, politics and oral proficiency: testing international air traffic controllers , 2009 .

[116]  Hyejeong Kim,et al.  Exploring the construct of aviation communication : a critique of the ICAO language proficiency policy , 2012 .

[117]  Henry Emery.  Developments in LSP Testing 30 Years On? The Case of Aviation English , 2014 .

[118]  Etienne Wenger,et al.  Situated Learning: Legitimate Peripheral Participation , 1991 .

[119]  J. Pill.  Drawing on indigenous criteria for more authentic assessment in a specific-purpose language test: Health professionals interacting with patients , 2016 .

[120]  T. McNamara.  Measuring Second Language Performance , 1996 .

[121]  Eugene Matusov,et al.  In Search of `the Appropriate' Unit of Analysis for Sociocultural Research , 2007 .

[122]  T. McNamara,et al.  Experimenting with Uncertainty: Essays in Honour of Alan Davies , 2001 .

[123]  Noam Chomsky,et al.  वाक्यविन्यास का सैद्धान्तिक पक्ष = Aspects of the theory of syntax , 1965 .

[124]  A. Cogo.  “They all Take the Risk and Make the Effort”: Intercultural Accommodation and Multilingualism in a BELF Community of Practice , 2016 .

[126]  C. Elder,et al.  Interrogating the construct of aviation English: Feedback from test takers in Korea , 2015 .

[128]  J. Greene.  Is Mixed Methods Social Inquiry a Distinctive Methodology? , 2008 .

[129]  Ana Lúcia T. Monteiro,et al.  Exploring Intercultural Factors in International Pilot-Air Traffic Controller Communications , 2018 .

[130]  Claire J. Kramsch.  From Language Proficiency to Interactional Competence , 1986 .

[131]  M. Irimia.  Developing the Intercultural Dimension in Language Teaching , 2012 .

[132]  Hong Wang,et al.  Traditional Literature Review and Research Synthesis , 2018 .

[133]  C. Elder,et al.  Interrogating the construct of communicative competence in language assessment contexts: What the non-language specialist can tell us , 2017 .

[134]  C. Weir.  Language Testing and Validation: An Evidence-Based Approach , 2004 .

[135]  R. Young.  Interactional Competence in Language Learning, Teaching, and Testing , 2011 .

[136]  Jiska Engelbert,et al.  Social construction of reality , 2016 .

[137]  W. Baker.  The Cultures of English as a Lingua Franca. , 2009 .

[138]  Samuel Messick,et al.  Validity and washback in language testing , 1996 .

[139]  U. Knoch.  Using subject specialists to validate an ESP rating scale: The case of the International Civil Aviation Organization (ICAO) rating scale , 2014 .

[140]  J. Creswell,et al.  Determining Validity in Qualitative Inquiry , 2000 .

[141]  Rapport DU Capscaafrica,et al.  INTERNATIONAL CIVIL AVIATION ORGANIZATION , 1947, International Organization.

[142]  M. Canale.  From communicative competence to communicative language pedagogy , 2014 .

[143]  Hyejeong Kim,et al.  Pronunciation and Comprehension in English as a Lingua Franca Communication: Effect of L1 Influence in International Aviation Communication , 2018 .