Topicalization in Language Models: A Case Study on Japanese

Proceedings of the 29th International Conference on Computational Linguistics, pages 851–862 October 12–17, 2022. 851 Topicalization in Language Models: A Case Study on Japanese Riki Fujihara1 Tatsuki Kuribayashi1,2 Kaori Abe1 Ryoko Tokuhisa1 Kentaro Inui1,3 1Tohoku University 2Langsmith Inc. 3RIKEN riki.fujihara.s4@dc.tohoku.ac.jp {kuribayashi, abe-k, tokuhisa, inui}@tohoku.ac.jp Abstract Humans use different wordings depending on the context to facilitate efficient communica- tion. For example, instead of completely new information, information related to the preceding context is typically placed at the sentence-initial position. In this study, we an- alyze whether neural language models (LMs) can capture such discourse-level preferences in text generation. Specifically, we focus on a particular aspect of discourse, namely the topic-comment structure. To analyze the linguistic knowledge of LMs separately, we chose the Japanese language, a topic- prominent language, for designing probing tasks, and we created human topicalization judgment data by crowdsourcing. Our experi- mental results suggest that LMs have different generalizations from humans; LMs exhibited less context-dependent behaviors toward top- icalization judgment. These results highlight the need for the additional inductive biases to guide LMs to achieve successful discourse- level generalization. 1 Introduction Building on the current success of neural language models (LMs) in the field of natural language pro- cessing (NLP), much work has been conducted to test their linguistic knowledge, typically, syntac- tic generalizations in LMs (Linzen et al., 2016; Lau et al., 2017; Marvin and Linzen, 2018; Gold- berg, 2019; Warstadt et al., 2019, 2020). Al- though discourse is also an essential aspect of lan- guage production along with syntactic construc- tions, discourse-level knowledge in LMs has been less explored or has been typically analyzed at the coarse level, for example, analyzing their sentence ordering abilities (Li and Jurafsky, 2017; See et al., 2019). As one step toward understanding the fine- grained, discourse-level knowledge in LMs, this study explores the generalization performance of 花瓶を昨日割った。 I broke a vase yesterday. 花瓶は部屋にあった。 The vase-TOP was in the room. (ii) Without Context (S+A) (i) With Context (C+S+A) The vase-{TOP/NOM} was in the room. I broke a vase yesterday. Humans LMs Task (to topicalize, or not to topicalize) compare preference TOP 𝑟!"#$% &'(') 𝑟*+ &'(') preference 𝑟!"#$% (') 𝑟*+ (') Corpus The vase-{TOP/NOM} was in the room. I broke a vase yesterday. NOM NOM NOM NOM TOP TOP TOP Figure 1: Comparing the context-dependent prefer- ences of language models (LMs) and humans for top- icalization. LMs in a particular aspect of discourse, the topic- comment structure (i.e., the thematic structure). The topic-comment structure is an essential part of language production (Grosz et al., 1995; Halli- day et al., 2014; Hajicová and Mírovsk`y, 2018). For example, speakers use the following sen- tences in different contexts: (1) a. In Japan, my father bought the vase last year. b. The vase was the one my father bought in Japan last year. These sentences differ in terms of topic-comment structure (the topic is underlined). For example, Sentence (1b) is more suited than (1a), for example, as a continuation of the sentence I broke a vase yes- …

Content truncated to 500 words