ConTextING: Granting Document-Wise Contextual Embeddings to Graph Neural Networks for Inductive Text Classification

Proceedings of the 29th International Conference on Computational Linguistics, pages 1–14 October 12–17, 2022. 1 Do language models make human-like predictions about the coreferents of Italian anaphoric zero pronouns? James A. Michaelov Department of Cognitive Science University of California, San Diego j1michae@ucsd.edu Benjamin K. Bergen Department of Cognitive Science University of California, San Diego bkbergen@ucsd.edu Abstract Some languages allow arguments to be omit- ted in certain contexts. Yet human language comprehenders reliably infer the intended ref- erents of these zero pronouns, in part because they construct expectations about which refer- ents are more likely. We ask whether Neural Language Models also extract the same expec- tations. We test whether 12 contemporary lan- guage models display expectations that reflect human behavior when exposed to sentences with zero pronouns from five behavioral experi- ments conducted in Italian by Carminati (2005). We find that three models—XGLM 2.9B, 4.5B, and 7.5B—capture the human behavior from all the experiments, with others successfully mod- eling some of the results. This result suggests that human expectations about coreference can be derived from exposure to language, and also indicates features of language models that al- low them to better reflect human behavior. 1 Introduction In Italian, like other pro-drop (‘pronoun-dropping’) languages, verbal arguments that would usually be expressed by pronouns in languages such as English can be omitted under certain circumstances. For example, consider the sentence in (1) from Carminati (2005). (1) Quando Maria ha chiamato Mario, era contenta. ‘When Maria called Mario, [she] was happy.’ In this sentence, the referent of the ‘dropped’ pronoun—generally referred to as a zero or null pronoun—can be inferred from the fact that the adjective contenta is feminine; thus, Maria is the most likely subject of the second clause. Resolv- ing the referents of anaphoric zero pronouns like this is a long-standing, important, and active area of research in natural language understanding (Ju- rafsky and Martin, 2021; for examples, see Zhao and Ng, 2007; Taira et al., 2008; Imamura et al., 2009; Watanabe et al., 2010; Kong and Zhou, 2010; Poesio et al., 2010; Chen and Ng, 2013; Yoshino et al., 2013; Iida et al., 2016; Aloraini and Poesio, 2020; Song et al., 2020a; Ueda et al., 2020; Konno et al., 2020, 2021; Yang et al., 2020; Kim et al., 2021; Umakoshi et al., 2021; Chen et al., 2021; Yang et al., 2022). It has been argued that aiming for human-like- ness in natural language processing systems is vital if we want our natural language understanding sys- tems to behave not only as humans do, but also as human users expect them to (see, e.g., Keller, 2010; Ettinger, 2020; Eisape et al., 2020). This is particularly true for zero anaphora resolution, and pronoun resolution more generally. As an illus- tration of the prominence of reference resolution, one pronoun resolution task, the Winograd Schema Challenge (Levesque et al., 2012; based on work by Winograd, 1972), has been referred to as ‘an al- ternative to the Turing Test’ (Levesque et al., 2012, p. 552). …

Content truncated to 500 words