Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP

Transfer learning, particularly approaches that combine multi-task learning with pre-trained contextualized embeddings and fine-tuning, have advanced the field of Natural Language Processing tremendously in recent years. In this paper we present MaChAmp, a toolkit for easy fine-tuning of contextualized embeddings in multi-task settings. The benefits of MaChAmp are its flexible configuration options, and the support of a variety of natural language processing tasks in a uniform toolkit, from text classification and sequence labeling to dependency parsing, masked language modeling, and text generation.

[1]  Mohammad Sadegh Rasooli,et al.  The Persian Dependency Treebank Made Universal , 2020, ArXiv.

[2]  Tunga Güngör,et al.  Resources for Turkish dependency parsing: introducing the BOUN Treebank and the BoAT annotation tool , 2020, Language Resources and Evaluation.

[3]  Challenges in Annotating and Parsing Spoken, Code-switched, Frisian-Dutch Data , 2021, ADAPTNLP.

[4]  Gihan Dias,et al.  ThamizhiUDp: A Dependency Parser for Tamil , 2020, ArXiv.

[5]  Ika Alfina,et al.  Tree Rotations for Dependency Trees: Converting the Head-Directionality of Noun Phrases , 2020 .

[6]  Antonio Toral,et al.  Character-level Representations Improve DRS-based Semantic Parsing Even in the Age of BERT , 2020, EMNLP.

[7]  Krister Lindén,et al.  Akkadian Treebank for early Neo-Assyrian Royal Inscriptions , 2020, TLT.

[8]  Kevin Duh,et al.  Very Deep Transformers for Neural Machine Translation , 2020, ArXiv.

[9]  Pooyan Fazli,et al.  Human-in-the-Loop Machine Learning to Increase Video Accessibility for Visually Impaired and Blind Users , 2020, Conference on Designing Interactive Systems.

[10]  Oliver Hellwig,et al.  The Treebank of Vedic Sanskrit , 2020, LREC.

[11]  Jennifer Foster,et al.  Treebank Embedding Vectors for Out-of-Domain Dependency Parsing , 2020, ACL.

[12]  Daniel Zeman,et al.  Universal Dependency Treebanks for Low-Resource Indian Languages: The Case of Bhojpuri , 2020, WILDRE.

[13]  Anton Karl Ingason,et al.  Creating a Parallel Icelandic Dependency Treebank from Raw Text to Universal Dependencies , 2020, LREC.

[14]  Olájídé Ishola,et al.  Yorùbá Dependency Treebank (YTB) , 2020, LREC.

[15]  Yulia Tsvetkov,et al.  Balancing Training for Multilingual Neural Machine Translation , 2020, ACL.

[16]  Orhan Firat,et al.  XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization , 2020, ICML.

[17]  Samuel R. Bowman,et al.  jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models , 2020, ACL.

[18]  Lysandre Debut,et al.  HuggingFace's Transformers: State-of-the-art Natural Language Processing , 2019, ArXiv.

[19]  Francis M. Tyers,et al.  Dependency annotation of noun incorporation in polysynthetic languages , 2020, UDW.

[20]  Daniel Zeman,et al.  Universal Dependencies for Albanian , 2020, UDW.

[21]  Kevin P. Scannell Universal Dependencies for Manx Gaelic , 2020, UDW.

[22]  Jack Rueter,et al.  On the questions in developing computational infrastructure for Komi-Permyak , 2020, IWCLUL.

[23]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[24]  Koichi Yasuoka Universal Dependencies Treebank of the Four Books in Classical Chinese , 2019 .

[25]  Zhou Yu,et al.  Dependency Parsing for Spoken Dialog Systems , 2019, EMNLP/IJCNLP.

[26]  Sylvain Kahane,et al.  A Surface-Syntactic UD Treebank for Naija , 2019, Proceedings of the 18th International Workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2019).

[27]  Cheikh M. Bamba Dione,et al.  Developing Universal Dependencies for Wolof , 2019, Proceedings of the Third Workshop on Universal Dependencies (UDW, SyntaxFest 2019).

[28]  Guillaume Thomas,et al.  Universal Dependencies for Mbyá Guaraní , 2019, Proceedings of the Third Workshop on Universal Dependencies (UDW, SyntaxFest 2019).

[29]  Özlem Çetinoğlu,et al.  Challenges of Annotating a Code-Switching Treebank , 2019 .

[30]  Niko Partanen,et al.  Survey of Uralic Universal Dependencies development , 2019, Proceedings of the Third Workshop on Universal Dependencies (UDW, SyntaxFest 2019).

[31]  Josef Ruppenhofer,et al.  tweeDe – A Universal Dependencies treebank for German tweets , 2019 .

[32]  Verginica Barbu Mititelu,et al.  MoNERo: a Biomedical Gold Standard Corpus for the Romanian Language , 2019, BioNLP@ACL.

[33]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[34]  Quoc V. Le,et al.  BAM! Born-Again Multi-Task Networks for Natural Language Understanding , 2019, ACL.

[35]  Daniel Kondratyuk,et al.  75 Languages, 1 Model: Parsing Universal Dependencies Universally , 2019, EMNLP.

[36]  Iain Murray,et al.  BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning , 2019, ICML.

[37]  Guillaume Lample,et al.  Cross-lingual Language Model Pretraining , 2019, NeurIPS.

[38]  Thomas Wolf,et al.  A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks , 2018, AAAI.

[39]  Samuel R. Bowman,et al.  Neural Network Acceptability Judgments , 2018, Transactions of the Association for Computational Linguistics.

[40]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[41]  Многоцелевой Морфологический,et al.  A REUSABLE TAGSET FOR THE MORPHOLOGICALLY RICH LANGUAGE IN CHANGE: A CASE OF MIDDLE RUSSIAN1 , 2019 .

[42]  Marie-Catherine de Marneffe,et al.  Conversion et améliorations de corpus du français annotés en Universal Dependencies [Conversion and Improvement of Universal Dependencies French corpora] , 2019, ICON.

[43]  Maximilian Wendt,et al.  HDT-UD: A very large Universal Dependencies Treebank for German , 2019, Proceedings of the Third Workshop on Universal Dependencies (UDW, SyntaxFest 2019).

[44]  Tommi A Pirinen,et al.  Building minority dependency treebanks, dictionaries and computational grammars at the same time—an experiment in Karelian treebanking , 2019, Proceedings of the Third Workshop on Universal Dependencies (UDW, SyntaxFest 2019).

[45]  Paolo Rosso,et al.  Presenting TWITTIRÒ-UD: An Italian Twitter Treebank in Universal Dependencies , 2019, Proceedings of the Fifth International Conference on Dependency Linguistics (Depling, SyntaxFest 2019).

[46]  Colin R. Batchelor Universal dependencies for Scottish Gaelic: syntax , 2019 .

[47]  Francis M. Tyers,et al.  Development of a Universal Dependencies treebank for Welsh , 2019 .

[48]  Ilya Sutskever,et al.  Language Models are Unsupervised Multitask Learners , 2019 .

[49]  Daniel Zeman,et al.  Data Conversion and Consistency of Monolingual Corpora: Russian UD Treebanks , 2018 .

[50]  Daniel Zeman,et al.  Challenges in Converting the Index Thomisticus Treebank into Universal Dependencies , 2018, UDW@EMNLP.

[51]  Francis M. Tyers,et al.  Multi-source synthetic treebank creation for improved cross-lingual dependency parsing , 2018, UDW@EMNLP.

[52]  Alina Wróblewska,et al.  Extended and Enhanced Polish Dependency Bank in Universal Dependencies Format , 2018, UDW@EMNLP.

[53]  Thierry Poibeau,et al.  The First Komi-Zyrian Universal Dependencies Treebanks , 2018, UDW@EMNLP.

[54]  Amir Zeldes,et al.  The Coptic Universal Dependency Treebank , 2018, UDW@EMNLP.

[55]  C. Aragon VARIAÇÕES ESTILÍSTICAS E SOCIAIS NO DISCURSO DOS FALANTES AKUNTSÚ , 2018 .

[56]  Adam Przepiórkowski,et al.  From Lexical Functional Grammar to enhanced Universal Dependencies , 2018, Language Resources and Evaluation.

[57]  Brendan T. O'Connor,et al.  Twitter Universal Dependency Parsing for African-American and Mainstream American English , 2018, ACL.

[58]  Guy Perrier,et al.  Application of Graph Rewriting to Natural Language Processing , 2018 .

[59]  Percy Liang,et al.  Know What You Don’t Know: Unanswerable Questions for SQuAD , 2018, ACL.

[60]  Na-Rae Han,et al.  Building Universal Dependency Treebanks in Korean , 2018, LREC.

[61]  Cristina Bosco,et al.  PoSTWITA-UD: an Italian Twitter Treebank in Universal Dependencies , 2018, LREC.

[62]  Yuji Matsumoto,et al.  Universal Dependencies Version 2 for Japanese , 2018, LREC.

[63]  Lilja Øvrelid,et al.  The LIA Treebank of Spoken Norwegian Dialects , 2018, LREC.

[64]  Peteris Paikens,et al.  Creation of a Balanced State-of-the-Art Multilayer Corpus for NLU , 2018, LREC.

[65]  Yusuke Miyao,et al.  Universal Dependencies for Amharic , 2018, LREC.

[66]  Joakim Nivre,et al.  Parser Training with Heterogeneous Treebanks , 2018, ACL.

[67]  Yijia Liu,et al.  Parsing Tweets into Universal Dependencies , 2018, NAACL.

[68]  Omer Levy,et al.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding , 2018, BlackboxNLP@EMNLP.

[69]  Barbara Plank,et al.  Strong Baselines for Neural Semi-Supervised Learning under Domain Shift , 2018, ACL.

[70]  Riyaz Ahmad Bhat,et al.  Universal Dependency Parsing for Hindi-English Code-Switching , 2018, NAACL.

[71]  Luke S. Zettlemoyer,et al.  AllenNLP: A Deep Semantic Natural Language Processing Platform , 2018, ArXiv.

[72]  Gerlof Bouma,et al.  The PROIEL treebank family: a standard for early attestations of Indo-European languages , 2017, Language Resources and Evaluation.

[73]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[74]  Sebastian Ruder,et al.  Universal Language Model Fine-tuning for Text Classification , 2018, ACL.

[75]  Simon Clematide,et al.  Parsing Approaches for Swiss German , 2018, SwissText.

[76]  Samuel R. Bowman,et al.  A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference , 2017, NAACL.

[77]  Francis M. Tyers,et al.  A prototype dependency treebank for Breton , 2018, CORIA-TALN-RJC.

[78]  Qiang Yang,et al.  An Overview of Multi-task Learning , 2018 .

[79]  Sanjiv Kumar,et al.  Adaptive Methods for Nonconvex Optimization , 2018, NeurIPS.

[80]  Gertjan van Noord,et al.  Modeling Input Uncertainty in Neural Network Dependency Parsing , 2018, EMNLP.

[81]  Milan Straka,et al.  UDPipe 2.0 Prototype at CoNLL 2018 UD Shared Task , 2018, CoNLL.

[82]  Taraka Rama,et al.  A Telugu treebank based on a grammar book , 2018, TLT.

[83]  Francis M. Tyers,et al.  Towards an open-source universal-dependency treebank for Erzya , 2018 .

[84]  Vinit Ravishankar,et al.  A Universal Dependencies Treebank for Marathi , 2018, TLT.

[85]  Daniel Zeman,et al.  Slovak Dependency Treebank in Universal Dependencies , 2017 .

[86]  John Lee,et al.  Quantitative Comparative Syntax on the Cantonese-Mandarin Parallel Dependency Treebank , 2017, DepLing.

[87]  Eckhard Bick,et al.  Universal Dependencies for Portuguese , 2017, DepLing.

[88]  Amir Zeldes,et al.  The GUM corpus: creating multilayer resources in the classroom , 2016, Language Resources and Evaluation.

[89]  Puneet Dwivedi,et al.  Universal Dependencies of Sanskrit , 2017 .

[90]  Eneko Agirre,et al.  SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation , 2017, *SEMEVAL.

[91]  Xavier Gómez Guinovart Recursos integrados da lingua galega para a investigación lingüística , 2017 .

[92]  Sebastian Ruder,et al.  An Overview of Multi-Task Learning in Deep Neural Networks , 2017, ArXiv.

[93]  Haris Papageorgiou,et al.  Universal Dependencies for Greek , 2017, UDW@NoDaLiDa.

[94]  Gertjan van Noord,et al.  Increasing Return on Annotation Investment: The Automatic Construction of a Universal Dependency Treebank for Dutch , 2017, UDW@NoDaLiDa.

[95]  Liesbeth Augustinus,et al.  Universal Dependencies for Afrikaans , 2017, UDW@NoDaLiDa.

[96]  John Lee,et al.  Towards Universal Dependencies for Learner Chinese , 2017, UDW@NoDaLiDa.

[97]  Mats Wirén,et al.  Universal Dependencies for Swedish Sign Language , 2017, NODALIDA.

[98]  Simon Krek,et al.  The Universal Dependencies Treebank for Slovenian , 2017, BSNLP@EACL.

[99]  Nikola Ljubesic,et al.  Universal Dependencies for Serbian in Comparison with Croatian and Other Slavic Languages , 2017, BSNLP@EACL.

[100]  Johan Bos,et al.  The Parallel Meaning Bank: Towards a Multilingual Corpus of Translations Annotated with Compositional Meaning Representations , 2017, EACL.

[101]  Barbara Plank,et al.  When is multitask learning effective? Semantic sequence prediction under varying data conditions , 2016, EACL.

[102]  Yoshimasa Tsuruoka,et al.  A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks , 2016, EMNLP.

[103]  Timothy Dozat,et al.  Deep Biaffine Attention for Neural Dependency Parsing , 2016, ICLR.

[104]  Moustapha Cissé,et al.  Efficient softmax approximation for GPUs , 2016, ICML.

[105]  Fei Xia,et al.  The Hindi/Urdu Treebank Project , 2017 .

[106]  Çağrı Çöltekin,et al.  A grammar-book treebank of Turkish , 2017 .

[107]  Francis M. Tyers,et al.  A Dependency Treebank for Kurmanji Kurdish , 2017, DepLing.

[108]  Tomas. Jelinek FicTree: A Manually Annotated Treebank of Czech Fiction , 2017, ITAT.

[109]  Francis M. Tyers,et al.  Annotation schemes in North Sámi dependency parsing , 2017 .

[110]  Francis M. Tyers,et al.  A Dependency Treebank for Buryat , 2017, TLT.

[111]  Benoît Sagot,et al.  From Noisy Questions to Minecraft Texts: Annotation Challenges in Extreme Syntax Scenario , 2016, NUT@COLING.

[112]  Yan Liu,et al.  Universal dependencies for Uyghur , 2016, WLSI/OIAF4HLT@COLING.

[113]  Çagri Çöltekin,et al.  Universal Dependencies for Turkish , 2016, COLING.

[114]  Cenel-Augusto Perez,et al.  Social Media - Processing Romanian Chat and Discourse Analysis , 2016, Computación y Sistemas.

[115]  George Kurian,et al.  Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation , 2016, ArXiv.

[116]  Daniel Zeman,et al.  Universal Dependencies for the AnCora treebanks , 2016, Proces. del Leng. Natural.

[117]  Anders Søgaard,et al.  Deep multi-task learning with low level tasks supervised at lower layers , 2016, ACL.

[118]  Barbara Plank,et al.  Multilingual Projection for Parsing Truly Low-Resource Languages , 2016, TACL.

[119]  Marie Candito,et al.  Hard Time Parsing Questions: Building a QuestionBank for French , 2016, LREC.

[120]  Boris Katz,et al.  Universal Dependencies for Learner English , 2016, ACL.

[121]  Mojgan Seraji,et al.  Universal Dependencies for Persian , 2016, LREC.

[122]  Lilja Øvrelid,et al.  Universal Dependencies for Norwegian , 2016, LREC.

[123]  Joakim Nivre,et al.  The Universal Dependencies Treebank of Spoken Slovenian , 2016, LREC.

[124]  Sampo Pyysalo,et al.  Universal Dependencies v1: A Multilingual Treebank Collection , 2016, LREC.

[125]  Barbara Plank,et al.  Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss , 2016, ACL.

[126]  Noah A. Smith,et al.  Many Languages, One Parser , 2016, TACL.

[127]  Rico Sennrich,et al.  Neural Machine Translation of Rare Words with Subword Units , 2015, ACL.

[128]  Jennifer Foster,et al.  Universal Dependencies for Irish , 2016 .

[129]  Fabio Tamburini,et al.  (Almost) Automatic Conversion of the Venice Italian Treebank into the Merged Italian Dependency Treebank Format , 2016, CLiC-it/EVALITA.

[130]  Lars Ahrenberg,et al.  Converting an English-Swedish Parallel Treebank to Universal Dependencies , 2015, DepLing.

[131]  Grzegorz Chrupala,et al.  Learning language through pictures , 2015, ACL.

[132]  Veronika Laippala,et al.  Universal Dependencies for Finnish , 2015, NODALIDA.

[133]  Johan Bos,et al.  Open-Domain Semantic Parsing with Boxer , 2015, NODALIDA.

[134]  Joakim Nivre,et al.  Towards a Universal Grammar for Natural Language Processing , 2015, CICLing.

[135]  Angeliki Lazaridou,et al.  Combining Language and Vision with a Multimodal Skip-gram Model , 2015, NAACL.

[136]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[137]  Héctor Martínez Alonso,et al.  Universal Dependencies for Danish , 2015 .

[138]  M. J. Aranzabe,et al.  Automatic Conversion of the Basque Dependency Treebank to Universal Dependencies , 2015 .

[139]  Nikola Ljubesic,et al.  Universal Dependencies for Croatian (that work for Serbian, too) , 2015, BSNLP@RANLP.

[140]  Simonetta Montemagni,et al.  The Evalita 2014 Dependency Parsing task , 2014 .

[141]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[142]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[143]  Cristina Bosco,et al.  Towards a Universal Stanford Dependenciesparallel treebank , 2014 .

[144]  Samuel R. Bowman,et al.  A Gold Standard Dependency Corpus for English , 2014, LREC.

[145]  Valentin Vydrin,et al.  Bamana Reference Corpus (BRC) , 2013 .

[146]  Christopher Potts,et al.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank , 2013, EMNLP.

[147]  Eiríkur Rögnvaldsson,et al.  The Icelandic Parsed Historical Corpus (IcePaHC) , 2012, LREC.

[148]  Zdenek Zabokrtský,et al.  Prague Dependency Style Treebank for Tamil , 2012, LREC.

[149]  Hector J. Levesque,et al.  The Winograd Schema Challenge , 2011, AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning.

[150]  Jason Weston,et al.  Natural Language Processing (Almost) from Scratch , 2011, J. Mach. Learn. Res..

[151]  David Bamman,et al.  The Ancient Greek and Latin Dependency Treebanks , 2011, Language Technology for Cultural Heritage.

[152]  Peter Clark,et al.  The Seventh PASCAL Recognizing Textual Entailment Challenge , 2011, TAC.

[153]  János Csirik,et al.  Hungarian Dependency Treebank , 2010, LREC.

[154]  Fei Xia,et al.  Hindi Syntax: Annotating Dependency, Lexical Predicate-Argument Structure, and Phrase Structure , 2009 .

[155]  Ido Dagan,et al.  The Sixth PASCAL Recognizing Textual Entailment Challenge , 2009, TAC.

[156]  Jan Hajic,et al.  The Czech Academic Corpus 2.0 Guide , 2008, Prague Bull. Math. Linguistics.

[157]  Marius L. Jøhndal,et al.  Creating a Parallel Treebank of the Old Indo-European BibleTranslations , 2008 .

[158]  Grzegorz Chrupala,et al.  Simple Data-Driven Context-Sensitive Lemmatization , 2006, Proces. del Leng. Natural.

[159]  Ido Dagan,et al.  The Third PASCAL Recognizing Textual Entailment Challenge , 2007, ACL-PASCAL@ACL.

[160]  Chris Brockett,et al.  Automatically Constructing a Corpus of Sentential Paraphrases , 2005, IJCNLP.

[161]  Wolfgang Lezius,et al.  TIGER: Linguistic Interpretation of a German Corpus , 2004 .

[162]  Petya Osenova,et al.  Design and Implementation of the Bulgarian HPSG-based Treebank , 2004 .

[163]  Rich Caruana,et al.  Multitask Learning , 1997, Machine Learning.

[164]  Erik F. Tjong Kim Sang,et al.  Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition , 2003, CoNLL.

[165]  Jan Hajic,et al.  The Prague Dependency Treebank , 2003 .

[166]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[167]  Danqi Chen,et al.  of the Association for Computational Linguistics: , 2001 .

[168]  Alexandra Kinyon,et al.  Building a Treebank for French , 2000, LREC.

[169]  Rich Caruana,et al.  Multitask Learning: A Knowledge-Based Source of Inductive Bias , 1993, ICML.