Harnessing the linguistic signal to predict scalar inferences

Language Models as an Alternative Evaluator of Word Order Hypotheses: A Case Study in Japanese Tatsuki Kuribayashi1,3, Takumi Ito1,3, Jun Suzuki1,2, Kentaro Inui1,2 1Tohoku University 2RIKEN 3Langsmith Inc. {kuribayashi, t-ito, jun.suzuki, inui} @ecei.tohoku.ac.jp Abstract We examine a methodology using neural lan- guage models (LMs) for analyzing the word order of language. This LM-based method has the potential to overcome the difficulties ex- isting methods face, such as the propagation of preprocessor errors in count-based meth- ods. In this study, we explore whether the LM- based method is valid for analyzing the word order. As a case study, this study focuses on Japanese due to its complex and flexible word order. To validate the LM-based method, we test (i) parallels between LMs and human word order preference, and (ii) consistency of the results obtained using the LM-based method with previous linguistic studies. Through our experiments, we tentatively conclude that LMs display sufficient word order knowledge for us- age as an analysis tool. Finally, using the LM- based method, we demonstrate the relationship between the canonical word order and topical- ization, which had yet to be analyzed by large- scale experiments. 1 Introduction Speakers sometimes have a range of options for word order in conveying a similar meaning. A typical case in English is dative alternation: (1) a. A teacher gave a student a book. b. A teacher gave a book to a student. Even for such a particular alternation, several stud- ies (Bresnan et al., 2007; Hovav and Levin, 2008; Colleman, 2009) investigated the factors determin- ing this word order and found that the choice is not random. For analyzing such linguistic phenom- ena, linguists repeat the cycle of constructing hy- potheses and testing their validity, usually through psychological experiments or count-based methods. However, these approaches sometimes face diffi- culties, such as scalability issues in psychological 0.0002 0.0000001 generation probabilities order1 is more likely. 質に quality-DAT effect-ACC gave. LM gave. effect-ACC quality-DAT (∅!""affected the quality.) order!: order": 影響を 与えた. 質に 影響を 与えた. Figure 1: LM-based method for evaluating the canoni- cality of each word order considering their generation probabilities. experiments and the propagation of preprocessor errors in count-based methods. Compared to the typical approaches for evaluat- ing linguistic hypotheses, approaches using LMs have potential advantages (Section 3.2). In this study, we examine the methodology of using LMs for analyzing word order (Figure 1). To validate the LM-based method, we first examine if there is a parallel between canonical word order and genera- tion probability of LMs for each word order. Futrell and Levy (2019) reported that English LMs have human-like word order preferences, which can be one piece of evidence for validating the LM-based method. However, it is not clear whether the above assumption is valid even in languages with more flexible word order. In this study, we specifically focus on the Japanese language due to its complex and flexible word order. There are many claims on the canonical word order of Japanese, and it has attracted consid- erable attention from …

Content truncated to 500 words