Modeling morphological learning, typology, and change: What can the neural sequence-to-sequence framework contribute?

We survey research using neural sequence-to-sequence models as compu- tational models of morphological learning and learnability. We discuss their use in determining the predictability of inflectional exponents, in making predictions about language acquisition and in modeling language change. Finally, we make some proposals for future work in these areas.

[1]  Ryan Cotterell,et al.  Modeling Word Forms Using Latent Underlying Morphs and Phonology , 2015, TACL.

[2]  Ryan Cotterell,et al.  On the Diachronic Stability of Irregularity in Inflectional Morphology , 2018, NAACL 2018.

[3]  John A. Goldsmith,et al.  Unsupervised Learning of the Morphology of a Natural Language , 2001, CL.

[4]  S. Pinker,et al.  On language and connectionism: Analysis of a parallel distributed processing model of language acquisition , 1988, Cognition.

[5]  Chris Dyer,et al.  Unsupervised Word Discovery with Segmental Neural Language Models , 2018, ArXiv.

[6]  Robert Malouf,et al.  Abstractive morphological learning with a recurrent neural network , 2017 .

[7]  Steven Pinker,et al.  Lexical semantics and irregular inflection , 2010, Language and cognitive processes.

[8]  J. Blevins.  Inflection Classes and Economy , 2004 .

[9]  David L. King,et al.  The OSU Realizer for SRST ‘18: Neural Sequence-to-Sequence Inflection and Incremental Locality-Based Linearization , 2018 .

[10]  Harry Bochner,et al.  Simplicity in Generative Morphology , 1993 .

[11]  J. Elman,et al.  Learning and morphological change , 1995, Cognition.

[12]  James L. McClelland,et al.  On learning the past-tenses of English verbs: implicit rules or parallel distributed processing , 1986 .

[13]  Michael Ramscar,et al.  The Zipfian Paradigm Cell Filling Problem , 2017 .

[14]  W. Dressler,et al.  On the role of morphological richness in the early development of noun and verb inflection , 2011 .

[15]  Aditi Lahiri,et al.  Analogy, levelling, markedness : principles of change in phonology and morphology , 2000 .

[16]  James P. Blevins,et al.  Word-based morphology , 2006, Journal of Linguistics.

[17]  Gregory Stump,et al.  Inflectional Morphology: Conclusions, extensions, and alternatives , 2001 .

[18]  Hinrich Schütze,et al.  Efficient Higher-Order CRFs for Morphological Tagging , 2013, EMNLP.

[19]  Dunja Jutronic.  Morphological changes in the urban vernacular of the city of Split , 2001 .

[20]  Andrew Damerell Carstairs,et al.  Allomorphy in inflexion , 2013 .

[21]  Matthew Baerman,et al.  Understanding and measuring morphological complexity , 2015 .

[22]  Amy Bidgood,et al.  The retreat from overgeneralization in child language acquisition: word learning, morphology, and verb argument structure. , 2013, Wiley interdisciplinary reviews. Cognitive science.

[23]  Royal Skousen,et al.  Analogical Modeling Of Language , 1989 .

[24]  Joyce McDonough,et al.  When segmentation helps: Implicative structure and morph boundaries in the Navajo verb , 2017 .

[25]  G. Boyé,et al.  Realistic data and paradigms: the paradigm cell finding problem , 2018, Morphology.

[26]  Raphael A. Finkel,et al.  Morphological Typology: From Word to Paradigm , 2013 .

[27]  A. Harris.  On the Explanation of Typologically Unusual Structures , 2008 .

[28]  Eleni Leontaridi,et al.  The study of language in its social context , 1996 .

[29]  Charles D. Yang.  Rage against the machine: Evaluation metrics in the 21st century , 2017 .

[30]  P. Matthews.  Inflectional Morphology: A Theoretical Study Based on Aspects of Latin Verb Conjugation , 1972 .

[31]  B. Joseph.  Children rule, or do they (as far as innovations are concerned)? , 2011, Bilingualism: Language and Cognition.

[32]  Xavier Laborda,et al.  The Oxford Handbook of the History of Linguistics , 2013 .

[33]  G. Corbett,et al.  Russian noun stress and network morphology , 1996 .

[34]  Ryan Cotterell,et al.  Weird Inflects but OK: Making Sense of Morphological Generation Errors , 2019, Conference on Computational Natural Language Learning.

[35]  Natalia Gagarina,et al.  On the Typology of Inflection Class Systems , 2006 .

[36]  B. Joseph.  Morphologization from Syntax , 2008 .

[37]  H. H. Hock.  Principles of historical linguistics , 1986 .

[38]  M. Baerman.  Covert systematicity in a distributionally complex system1 , 2013, Journal of Linguistics.

[39]  Ryan Cotterell,et al.  On the Complexity and Typology of Inflectional Morphological Systems , 2018, TACL.

[40]  Rachel Nordlinger,et al.  The Acquisition of Murrinhpatha (Northern Australia) , 2017 .

[41]  Stephen R. Anderson,et al.  Morphological universals and diachrony , 2005 .

[42]  B. Joseph,et al.  Language History, Language Change, and Language Relationship: An Introduction to Historical and Comparative Linguistics , 1996 .

[43]  R. Baayen,et al.  Putting the bits together: an information theoretical perspective on morphological processing , 2004, Cognition.

[44]  Ryan Cotterell,et al.  Neural Multi-Source Morphological Reinflection , 2016, EACL.

[45]  Gary Lupyan,et al.  Understanding the Origins of Morphological Diversity: the Linguistic Niche Hypothesis , 2012, Adv. Complex Syst..

[46]  Fermín Moscoso del Prado Martín,et al.  The simultaneous effects of inflectional paradigms and classes on lexical recognition: Evidence from Serbian , 2009 .

[47]  J. Greenberg.  A Quantitative Approach to the Morphological Typology of Language , 1960, International Journal of American Linguistics.

[48]  S. Strogatz,et al.  Linguistics: Modelling the dynamics of language death , 2003, Nature.

[49]  T. Griffiths,et al.  Iterated learning: Intergenerational knowledge transmission reveals inductive biases , 2007, Psychonomic bulletin & review.

[50]  A. Carstairs.  The evolution of morphology , 2010 .

[51]  H. Harley,et al.  DISTRIBUTED MORPHOLOGY , 1999 .

[52]  S. Goldwater,et al.  Are we there yet? Encoder-decoder neural networks as cognitive models of English past tense inflection , 2019, ACL.

[53]  M. Aronoff.  Morphology by Itself: Stems and Inflectional Classes , 1993 .

[54]  Jeff Parker.  Inflectional Complexity and Cognitive Processing: An Experimental and Corpus-based Investigation of Russian Nouns , 2016 .

[55]  Maxi San Miguel,et al.  Agent-based models of language competition , 2013 .

[56]  Yoav Goldberg,et al.  Morphological Inflection Generation with Hard Monotonic Attention , 2016, ACL.

[57]  Olivier Bonami,et al.  Joint predictiveness in inflectional paradigms , 2016 .

[58]  P. Kiparsky.  Linguistic universals and linguistic change , 1968 .

[59]  Grzegorz Kondrak,et al.  Inflection Generation as Discriminative String Transduction , 2015, HLT-NAACL.

[60]  Wolfgang Ullrich Wurzel,et al.  Inflectional Morphology and Naturalness , 1989 .

[61]  E. Dąbrowska.  The effects of frequency and neighbourhood density on adult speakers’ productivity with Polish case inflections: an empirical test of usage-based approaches to morphology , 2008 .

[62]  Yulia Tsvetkov,et al.  Morphological Inflection Generation Using Character Sequence to Sequence Learning , 2015, NAACL.

[63]  Markus Dreyer,et al.  Discovering Morphological Paradigms from Plain Text Using a Dirichlet Process Mixture Model , 2011, EMNLP.

[64]  Ling Liu,et al.  A Computational Model for the Linguistic Notion of Morphological Paradigm , 2018, COLING.

[65]  S. Kirby,et al.  The emergence of linguistic structure: an overview of the iterated learning model , 2002 .

[66]  Peter Tiersma,et al.  Local and General Markedness , 1982 .

[67]  Joan L. Bybee.  Diachronic and typological properties of morphology and their implications for representation , 1995 .

[68]  Graham Neubig,et al.  Unsupervised Learning of Syntactic Structure with Invertible Neural Projections , 2018, EMNLP.

[69]  Raphael A. Finkel,et al.  Principal parts and degrees of paradigmatic transparency , 2009 .

[70]  James R. Hurford,et al.  Learning, Culture and Evolution in the Origin of Linguistic Constraints , 1997 .

[71]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[72]  Joe Pater,et al.  Seq2Seq Models with Dropout can Learn Generalizable Reduplication , 2018 .

[73]  Bruce Hayes,et al.  Modeling English Past Tense Intuitions with Minimal Generalization , 2002, SIGMORPHON.

[74]  Alec Marantz,et al.  Clarifying “blur”: Paradigms, defaults, and inflectional classes , 2008 .

[75]  Yoav Seginer,et al.  Fast Unsupervised Incremental Parsing , 2007, ACL.

[76]  Mark Steedman,et al.  Two Decades of Unsupervised POS Induction: How Far Have We Come? , 2010, EMNLP.

[77]  M. Maratsos.  More overregularizations after all: new data and discussion on Marcus, Pinker, Ullman, Hollander, Rosen & Xu , 2000, Journal of Child Language.

[78]  H. Gleitman,et al.  Human simulations of vocabulary learning , 1999, Cognition.

[79]  R. Baayen,et al.  The balance of storage and computation in morphological processing: the role of word formation type, affixal homonymy, and productivity. , 2000, Journal of experimental psychology. Learning, memory, and cognition.

[80]  Eric Gaussier,et al.  Unsupervised learning of derivational morphology from inflectional lexicons , 1999 .

[81]  Kenny Smith,et al.  A model of linguistic accommodation leading to language simplification , 2018, CogSci.

[82]  Gregory Shakhnarovich,et al.  Visually Grounded Learning of Keyword Prediction from Untranscribed Speech , 2017, INTERSPEECH.

[83]  Robert Malouf.  The No Blur Principle Effects as an Emergent Property of Language Systems , 2015 .

[84]  Morris Halle,et al.  Distributed morphology and the pieces of inflection , 1993 .

[85]  Carol Lynn Moder,et al.  Morphological Classes as Natural Categories , 1983 .

[86]  Joan L. Bybee,et al.  Mechanisms of Change in Grammaticization: The Role of Frequency , 2008 .

[87]  John DeNero,et al.  Supervised Learning of Complete Morphological Paradigms , 2013, NAACL.

[88]  Thomas L. Griffiths,et al.  Interpolating between types and tokens by estimating power-law generators , 2005, NIPS.

[89]  Olivier Bonami,et al.  Inferring Inflection Classes with Description Length , 2018, J. Lang. Model..

[90]  Andrea D. Sims,et al.  How inflection class systems work: On the informativity of implicative structure , 2016 .

[91]  J. Berko.  The Child's Learning of English Morphology , 1958 .

[92]  Claudia Marzi,et al.  Computational complexity of abstractive morphology , 2015 .

[93]  Lane Schwartz,et al.  Unsupervised Grammar Induction with Depth-bounded PCFG , 2018, TACL.

[94]  A. Carstairs-McCarthy.  Inflection classes, gender, and the principle of contrast , 1994 .

[95]  R. Finkel,et al.  The complexity of inflectional systems , 2015 .

[96]  Eve V. Clark,et al.  The principle of contrast: A constraint on language acquisition. , 1987 .

[97]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[98]  S. Kirby,et al.  The emergence of linguistic structure: an overview of the iterated learning model , 2002 .

[99]  L. Gerken,et al.  Analogy in Grammar: Form and Acquisition , 2006 .

[100]  Andrea D. Sims,et al.  Interpreting Sequence-to-Sequence Models for Russian Inflectional Morphology , 2020, SCIL.

[101]  Kallirroi Georgila,et al.  Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08) , 2008 .

[102]  Charles D. Yang,et al.  Morphology and Language Acquisition , 2016 .

[103]  Lesley Milroy,et al.  Off the shelf or under the counter? On the social dynamics of sound changes , 2007 .

[104]  Josef van Genabith,et al.  Learning Morphology with Morfette , 2008, LREC.

[105]  R. H. Baayen,et al.  Storage and computation in the mental lexicon , 2005 .

[106]  L. Gleitman,et al.  When we think about thinking: The acquisition of belief verbs , 2007, Cognition.

[107]  Thomas L. Griffiths,et al.  The evolution of frequency distributions: Relating regularization to inductive biases through iterated learning , 2009, Cognition.

[108]  Adam Albright.  Islands of Reliability for Regular Morphology: Evidence from Italian , 2003 .

[109]  Geoffrey Sampson,et al.  Word frequency distributions , 2002, Computational Linguistics.

[110]  Nancy C. Dorian.  The fate of morphological complexity in language death: Evidence from East Sutherland Gaelic , 1978 .

[111]  Stephen R. Anderson,et al.  A-Morphous morphology , 1992 .

[112]  L. Feldman.  Diachronic and Typological Properties of Morphology and Their Implications for Representation , 2013 .

[113]  Dan Klein,et al.  Learning Accurate, Compact, and Interpretable Tree Annotation , 2006, ACL.

[114]  Sampo Pyysalo,et al.  Universal Dependencies v1: A Multilingual Treebank Collection , 2016, LREC.

[115]  Dunstan Brown,et al.  Network morphology : a defaults-based theory of word structure , 2012 .

[116]  C. Jamieson.  Conflated Subsystems Marking Person and Aspect in Chiquihuitlán Mazatec Verbs , 1982, International Journal of American Linguistics.

[117]  W. Labov.  Transmission and Diffusion , 2007 .

[118]  C. Lefebvre.  The Tense, Mood, and Aspect System of Haitian Creole and the Problem of Transmission of Grammar in Creole Genesis , 1996 .

[119]  Martin Maiden,et al.  Morphological autonomy and diachrony , 2005 .

[120]  Katharina Kann,et al.  MED: The LMU System for the SIGMORPHON 2016 Shared Task on Morphological Reinflection , 2016, SIGMORPHON.

[121]  James P. Blevins.  Word and Paradigm Morphology , 2016 .

[122]  A. E. Albright,et al.  The identification of bases in morphological paradigms , 2002 .

[123]  Haukur Þorgeirsson.  Testing Vocabular Clarity in insular Scandinavian , 2017 .

[124]  Ryan Cotterell,et al.  The SIGMORPHON 2016 Shared Task—Morphological Reinflection , 2016, SIGMORPHON.

[125]  P. Trudgill.  Sociolinguistic Typology: Social Determinants of Linguistic Complexity , 2011 .

[126]  Dunstan Brown,et al.  Frequency, Regularity, and the Paradigm: A Perspective from Russian on a Complex Relation , 2001 .

[127]  Robert Malouf,et al.  Morphological Organization: The Low Conditional Entropy Conjecture , 2013 .

[128]  Thomas L. Griffiths,et al.  Adaptor Grammars: A Framework for Specifying Compositional Nonparametric Bayesian Models , 2006, NIPS.