Are representations built from the ground up? An empirical examination of local composition in language models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9053 - 9073 December 7-11, 2022 ©2022 Association for Computational Linguistics Are Representations Built from the Ground Up? An Empirical Examination of Local Composition in Language Models Emmy Liu and Graham Neubig Language Technologies Institute Carnegie Mellon University {mengyan3, gneubig}@cs.cmu.edu Abstract Compositionality, the phenomenon where the meaning of a phrase can be derived from its constituent parts, is a hallmark of human lan- guage. At the same time, many phrases are non-compositional, carrying a meaning beyond that of each part in isolation. Representing both of these types of phrases is critical for lan- guage understanding, but it is an open question whether modern language models (LMs) learn to do so; in this work we examine this question. We first formulate a problem of predicting the LM-internal representations of longer phrases given those of their constituents. We find that the representation of a parent phrase can be predicted with some accuracy given an affine transformation of its children. While we would expect the predictive accuracy to correlate with human judgments of semantic compositionality, we find this is largely not the case, indicating that LMs may not accurately distinguish be- tween compositional and non-compositional phrases. We perform a variety of analyses, shedding light on when different varieties of LMs do and do not generate compositional rep- resentations, and discuss implications for future modeling work.1 1 Introduction Compositionality is argued to be a hallmark of lin- guistic generalization (Szabó, 2020). However, some phrases are non-compositional, and can- not be reconstructed from individual constituents (Dankers et al., 2022a). Intuitively, a phrase like "I own cats and dogs" is locally compositional, whereas "It’s raining cats and dogs" is not. There- fore, any representation of language must be easily composable, but it must also correctly handle cases that deviate from compositional rules. Both lack (Hupkes et al., 2020; Lake and Baroni, 2017) and excess (Dankers et al., 2022b) of compo- sitionality have been cited as common sources of 1Code and data available at https://github.com/ nightingal3/lm-compositionality X A B [CLS] the dog sits on the sofa [SEP] [CLS] the dog [SEP] [CLS] sits on the sofa [SEP] Figure 1: An illustration of the local composition pre- diction problem with [CLS] representations. errors in NLP models, indicating that models may handle phrase composition in an unexpected way. In general form, the compositionality principle is simply “the meaning of an expression is a func- tion of the meanings of its parts and of the way they are syntactically combined” (Pelletier, 1994). However, this definition is underspecified (Partee, 1984). Recent efforts to evaluate the compositional abilities of neural networks have resulted in several testable definitions of compositionality (Hupkes et al., 2020). Previous work on compositionality in natural language focuses largely on the definition of substi- tutivity, by focusing on changes to the constituents of a complex phrase and how they change its repre- sentation (Dankers et al., 2022a; Garcia et al., 2021; Yu and Ettinger, 2020). The …

Content truncated to 500 words