Call Larisa Ivanovna: Code-Switching Fools Multilingual NLU Models

Practical needs of developing task-oriented dialogue assistants require the ability to understand many languages. Novel benchmarks for multilingual natural language understanding (NLU) include monolingual sentences in several languages, annotated with intents and slots. In such setup models for cross-lingual transfer show remarkable performance in joint intent recognition and slot filling. However, existing benchmarks lack of code-switched utterances, which are difficult to gather and label due to complexity in the grammatical structure. The evaluation of NLU models seems biased and limited, since code-switching is being left out of scope. Our work adopts recognized methods to generate plausible and naturally-sounding code-switched utterances and uses them to create a synthetic code-switched test set. Based on experiments, we report that the state-of-the-art NLU models are unable to handle code-switching. At worst, the performance, evaluated by semantic accuracy, drops as low as 15% from 80% across languages. Further we show, that pre-training on synthetic code-mixed data helps to maintain performance on the proposed test set at a comparable level with monolingual data. Finally, we analyze different language pairs and show that the closer the languages are, the better the NLU model handles their alternation. This is in line with the common understanding of how multilingual models conduct transferring between languages.

[1]  Xipeng Qiu,et al.  BERT-ATTACK: Adversarial Attack against BERT Using BERT , 2020, EMNLP.

[2]  M. Choudhury,et al.  BERTologiCoMix: How does Code-Mixing interact with Multilingual BERT? , 2021, ADAPTNLP.

[3]  Thamar Solorio,et al.  LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation , 2020, LREC.

[4]  Haizhou Li,et al.  Modeling Code-Switch Languages Using Bilingual Parallel Corpus , 2020, ACL.

[5]  Wen Wang,et al.  BERT for Joint Intent Classification and Slot Filling , 2019, ArXiv.

[6]  Dejing Dou,et al.  HotFlip: White-Box Adversarial Examples for Text Classification , 2017, ACL.

[7]  Monojit Choudhury,et al.  GLUECoS: An Evaluation Benchmark for Code-Switched NLP , 2020, ACL.

[8]  Pascale Fung,et al.  Cross-lingual Spoken Language Understanding with Regularized Representation Alignment , 2020, EMNLP.

[9]  Huzefa Rangwala,et al.  Multilingual Code-Switching for Zero-Shot Cross-Lingual Intent Prediction and Slot Filling , 2021, MRL.

[10]  John X. Morris,et al.  TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP , 2020, EMNLP.

[11]  Tanuja Ganu,et al.  GCM: A Toolkit for Generating Synthetic Code-mixed Text , 2021, EACL.

[12]  Pascale Fung,et al.  Code-Switched Language Models Using Neural Based Synthetic Data from Parallel Sentences , 2019, CoNLL.

[13]  Pascale Fung,et al.  Are Multilingual Models Effective in Code-Switching? , 2021, CALCS.

[14]  Marjan Ghazvininejad,et al.  Multilingual Denoising Pre-training for Neural Machine Translation , 2020, Transactions of the Association for Computational Linguistics.

[15]  Jinfeng Yi,et al.  Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples , 2018, AAAI.

[16]  Ponnurangam Kumaraguru,et al.  CoMeT: Towards Code-Mixed Translation Using Parallel Monolingual Sentences , 2021, CALCS.

[17]  Barbara Plank,et al.  From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding , 2021, NAACL.

[18]  David A. van Leeuwen,et al.  Acoustic and Textual Data Augmentation for Improved ASR of Code-Switching Speech , 2018, INTERSPEECH.

[19]  Ignacio Iacobacci,et al.  XeroAlign: Zero-shot cross-lingual transformer alignment , 2021, FINDINGS.

[20]  Peng Xu,et al.  Attention-Informed Mixed-Language Training for Zero-shot Cross-lingual Task-oriented Dialogue Systems , 2019, AAAI.

[21]  Graham Neubig,et al.  XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization , 2020, ICML.

[22]  Anna Korhonen,et al.  Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems , 2021, J. Artif. Intell. Res..

[23]  David Sankoff,et al.  A Formal Grammar for Code-Switching. CENTRO Working Papers 8. , 1980 .

[24]  Gökhan Tür,et al.  What is left to be understood in ATIS? , 2010, 2010 IEEE Spoken Language Technology Workshop.

[25]  Richard Socher,et al.  XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering , 2019, ArXiv.

[26]  Muhammad Abdul-Mageed,et al.  Exploring Text-to-Text Transformers for English to Hinglish Machine Translation with Synthetic Code-Mixing , 2021, CALCS.

[27]  Abhinav Arora,et al.  El Volumen Louder Por Favor: Code-switching in Task-oriented Semantic Parsing , 2021, EACL.

[28]  Libo Qin,et al.  CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP , 2020, ArXiv.

[29]  Niloy Ganguly,et al.  A Deep Generative Model for Code-Switched Text , 2019, IJCAI.

[30]  Saab Mansour,et al.  End-to-End Slot Alignment and Recognition for Cross-Lingual NLU , 2020, EMNLP.

[31]  Antonio Jimeno-Yepes,et al.  Grey-box Adversarial Attack And Defence For Sentiment Classification , 2021, NAACL.

[32]  Shafiq Joty,et al.  Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots , 2021, NAACL.

[33]  Sunita Sarawagi,et al.  Training Data Augmentation for Code-Mixed Translation , 2021, NAACL.

[34]  Graham Neubig,et al.  Word Alignment by Fine-tuning Embeddings on Parallel Corpora , 2021, EACL.

[35]  Thamar Solorio,et al.  Language Identification and Analysis of Code-Switched Social Media Text , 2018, CodeSwitch@ACL.

[36]  Yanjun Qi,et al.  Black-Box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers , 2018, 2018 IEEE Security and Privacy Workshops (SPW).

[37]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[38]  Jian Pei,et al.  Learning from Multiple Noisy Augmented Data Sets for Better Cross-Lingual Spoken Language Understanding , 2021, EMNLP.

[39]  Haoran Li,et al.  MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark , 2020, EACL.

[40]  Veselin Stoyanov,et al.  Unsupervised Cross-lingual Representation Learning at Scale , 2019, ACL.

[41]  Holger Schwenk,et al.  Beyond English-Centric Multilingual Machine Translation , 2020, J. Mach. Learn. Res..

[42]  Yasemin Altun,et al.  Translate & Fill: Improving Zero-Shot Multilingual Semantic Parsing with Synthetic Data , 2021, EMNLP.

[43]  Young-Bum Kim,et al.  Efficient Large-Scale Domain Classification with Personalized Attention , 2018, ArXiv.

[44]  Richard Socher,et al.  TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue , 2020, EMNLP.

[45]  Radhika Mamidi,et al.  Political Discourse Analysis: A Case Study of Code Mixing and Code Switching in Political Speeches , 2021, CALCS.