Implicit Representations of Meaning in Neural Language Models

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1813–1827 August 1–6, 2021. ©2021 Association for Computational Linguistics 1813 Implicit Representations of Meaning in Neural Language Models Belinda Z. Li Maxwell Nye Jacob Andreas Massachusetts Institute of Technology {bzl,mnye,jda}@mit.edu Abstract Does the effectiveness of neural language mod- els derive entirely from accurate modeling of surface word co-occurrence statistics, or do these models represent and reason about the world they describe? In BART and T5 trans- former language models, we identify contex- tual word representations that function as mod- els of entities and situations as they evolve throughout a discourse. These neural represen- tations have functional similarities to linguis- tic models of dynamic semantics: they support a linear readout of each entity’s current prop- erties and relations, and can be manipulated with predictable effects on language genera- tion. Our results indicate that prediction in pre- trained neural language models is supported, at least in part, by dynamic representations of meaning and implicit simulation of entity state, and that this behavior can be learned with only text as training data.1 1 Introduction Neural language models (NLMs), which place probability distributions over sequences of words, produce contextual word and sentence embeddings that are useful for a variety of language process- ing tasks (Peters et al., 2018; Lewis et al., 2020). This usefulness is partially explained by the fact that NLM representations encode lexical relations (Mikolov et al., 2013) and syntactic structure (Ten- ney et al., 2019). But the extent to which NLM training also induces representations of meaning remains a topic of ongoing debate (Bender and Koller, 2020; Wu et al., 2021). In this paper, we show that NLMs represent meaning in a specific sense: in simple semantic domains, they build rep- resentations of situations and entities that encode logical descriptions of each entity’s dynamic state. 1Code and data are available at https://github.com/ belindal/state-probes. You see an open chest. The only thing in the chest is an old key. There is a locked wooden door leading east. Next, you… (c1) …use the key to unlock the door. (c2) …drop an apple on the ground. (c3) …remove an apple from the chest. You pick up the key. (a) chest open contains key you (a )′ (b) LM encoder semantic probe LM decoder (b )′ door locked chest open possesses key you door locked empty Figure 1: Neural language models trained on text alone (a–c) produce semantic representations that en- code properties and relations of entities mentioned in a discourse (a′). Representations are updated when the discourse describes changes to entities’ state (b′). Consider the text in the left column of Fig. 1. Sentences (a) describe the contents of a room; this situation can be formally characterized by the graph of entities, properties, and relations depicted in (a′). Sentence (b), You pick up the key, causes the situa- tion to change: a chest becomes empty, and a key becomes possessed by …

Content truncated to 500 words