Language Models as an Alternative Evaluator of Word Order Hypotheses: A Case Study in Japanese

We examine a methodology using neural language models (LMs) for analyzing the word order of language. This LM-based method has the potential to overcome the difficulties existing methods face, such as the propagation of preprocessor errors in count-based methods. In this study, we explore whether the LM-based method is valid for analyzing the word order. As a case study, this study focuses on Japanese due to its complex and flexible word order. To validate the LM-based method, we test (i) parallels between LMs and human word order preference, and (ii) consistency of the results obtained using the LM-based method with previous linguistic studies. Through our experiments, we tentatively conclude that LMs display sufficient word order knowledge for usage as an analysis tool. Finally, using the LM-based method, we demonstrate the relationship between the canonical word order and topicalization, which had yet to be analyzed by large-scale experiments.

[1]  I. A. Belmonte.  Teaching English Word Order to ESL Spanish Students. A Functional Perspective , 1999 .

[2]  Timothy Colleman,et al.  Verb disposition in argument structure alternations: a corpus study of the dative alternation in Dutch☆ , 2009 .

[3]  Hajime Hoji,et al.  Logical form constraints and configurational structures in Japanese , 1985 .

[4]  Taku Kudo,et al.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing , 2018, EMNLP.

[5]  N. Akhtar,et al.  Acquiring basic word order: evidence for data-driven learning of syntactic structure , 1999, Journal of Child Language.

[6]  Richard Futrell,et al.  Do RNNs learn human-like abstract word order preferences? , 2018, ArXiv.

[7]  Masatoshi Koizumi,et al.  Cognitive Processing of Japanese Sentences with Ditransitive Verbs , 2004 .

[8]  Taku Kudo,et al.  MeCab : Yet Another Part-of-Speech and Morphological Analyzer , 2005 .

[9]  Hsin-Hsi Chen,et al.  Chinese Word Ordering Errors Detection and Correction for Non-Native Chinese Language Learners , 2014, COLING.

[10]  Katharina Kann,et al.  Sentence-Level Fluency Evaluation: References Help, But Can Be Spared! , 2018, CoNLL.

[11]  Caroline Heycock,et al.  Syntactic predication in Japanese , 1993 .

[12]  Dan I. Moldovan,et al.  Language Models and Reranking for Machine Translation , 2006, WMT@HLT-NAACL.

[13]  Karin Harbusch,et al.  A corpus study into word order variation in German subordinate clauses: Animacy affects linearization independently of function assignment , 2003 .

[14]  Natsuko Tsujimura,et al.  An Introduction to Japanese Linguistics , 1997 .

[15]  R. Harald Baayen,et al.  Predicting the dative alternation , 2007 .

[16]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[17]  Moustapha Cissé,et al.  Efficient softmax approximation for GPUs , 2016, ICML.

[18]  Malka Rappaport Hovav,et al.  The English dative alternation: The case for verb sensitivity1 , 2008, Journal of Linguistics.

[19]  N. Snider,et al.  More than words: Frequency effects for multi-word phrases , 2010 .

[20]  Naho Orita.  Predicting Japanese scrambling in the wild , 2017, CMCL@EACL.

[21]  Jelke Bloem,et al.  Testing the Processing Hypothesis of word order variation using a probabilistic language model , 2016, CL4LC@COLING 2016.

[22]  Yasumasa Shigenaga.  Canonical Word Order of Japanese Ditransitive Sentences: A Preliminary Investigation through a Grammaticality Judgment Survey , 2014 .

[23]  Masayuki Asahara,et al.  Predicting Japanese Word Order in Double Object Constructions , 2018 .

[24]  Rico Sennrich,et al.  Neural Machine Translation of Rare Words with Subword Units , 2015, ACL.

[25]  Manabu Okumura,et al.  A Corpus-Based Analysis of Canonical Word Order of Japanese Double Object Constructions , 2016, ACL.

[26]  E. T. Miyamoto.  Sources of difficulty in processing scrambling in Japanese , 2002 .

[27]  Thomas F Münte,et al.  An fMRI study of canonical and noncanonical word order in German , 2007, Human brain mapping.

[28]  Natsuko Nakagawa,et al.  Information structure in spoken Japanese : Particles, word order, and intonation , 2020 .

[29]  Myle Ott,et al.  fairseq: A Fast, Extensible Toolkit for Sequence Modeling , 2019, NAACL.

[30]  Ananthakrishnan Ramanathan,et al.  A Word Reordering Model for Improved Machine Translation , 2011, EMNLP.

[31]  D. Slobin,et al.  Children use canonical sentence schemas: A crosslinguistic study of word order and inflections , 1982, Cognition.

[32]  J. Fletcher.  Distribution , 2009, BMJ : British Medical Journal.

[33]  John Fry,et al.  Ellipsis and wa-marking in Japanese Conversation , 2001 .

[34]  Mikinari Matsuoka,et al.  Two Types of Ditransitive Consturctions in Japanese , 2003 .

[35]  Bernard Comrie,et al.  Language Universals and Linguistic Typology: Syntax and Morphology , 1981 .