New or Old? Exploring How Pre-Trained Language Models Represent Discourse Entities

Proceedings of the 29th International Conference on Computational Linguistics, pages 875–886 October 12–17, 2022. 875 New or Old? Exploring How Pre-Trained Language Models Represent Discourse Entities Sharid Loáiciga∗1, Anne Beyer∗2, David Schlangen2 1CLASP, Dept. of Philosophy, Linguistics and Theory of Science, University of Gothenburg 2Computational Linguistics, Department of Linguistics University of Potsdam, Germany sharid.loaiciga@gu.se anne.beyer,david.schlangen@uni-potsdam.de Abstract Recent research shows that pre-trained lan- guage models, built to generate text condi- tioned on some context, learn to encode syn- tactic knowledge to a certain degree. This has motivated researchers to move beyond the sentence-level and look into their ability to en- code less studied discourse-level phenomena. In this paper, we add to the body of probing research by investigating discourse entity repre- sentations in large pre-trained language models in English. Motivated by early theories of dis- course and key pieces of previous work, we focus on the information-status of entities as discourse-new or discourse-old. We present two probing models, one based on binary clas- sification and another one on sequence labeling. The results of our experiments show that pre- trained language models do encode information on whether an entity has been introduced before or not in the discourse. However, this informa- tion alone is not sufficient to find the entities in a discourse, opening up interesting questions about the definition of entities for future work. 1 Introduction In a seminal paper from 1969, Karttunen imagines “a device designed to read a text in some natural language, interpret it, and store the content in some manner, say, for the purpose of being able to answer questions about it”. Such a device—considered by him “not a practical idea, for the time being at least”—he says would need to have a particular feature, namely that it “be able to recognize when a novel individual is mentioned in the input text and to store it along with its characterization for future reference.” Now, more than 50 years later, neural models appear to have made such a device a practical idea after all. But do they recognize when a text intro- duces a new entity into the “universe of discourse”, or when, in contrast, the new information concerns ∗Shared first authorship. a previously introduced one? This is the question that we are asking in this paper. Following Karttunen’s idea and inspired by Prince (1992)’s analysis of information-status, we focus on discourse entities. In particular, we tar- get the task of distinguishing between the status of entity mentions as discourse-new or discourse-old. Considering that discourse entities are central to discourse theories and meaning, we consider that this is an understudied subject in the field. We take a step back from much more specific tasks such as coreference resolution and look at entities being referred to over time. We believe that the new/old distinction, as a simplified form of discourse repre- sentation, lets us ask whether language models are able to keep track of discourse entities. Concretely, we build probing models that take as input the representations of pre-trained …

Content truncated to 500 words