Transparency Helps Reveal When Language Models Learn Meaning

Transparency Helps Reveal When Language Models Learn Meaning Zhaofeng Wu ∗ William Merrill Hao Peng Iz Beltagy Noah A. Smith MIT New York University Allen Institute for Artificial Intelligence Paul G. Allen School of Computer Science & Engineering, University of Washington zfw@csail.mit.edu willm@nyu.edu {haop,beltagy,noah}@allenai.org Abstract Many current NLP systems are built from language models trained to optimize unsu- pervised objectives on large amounts of raw text. Under what conditions might such a pro- cedure acquire meaning? Our systematic ex- periments with synthetic data reveal that, with languages where all expressions have context- independent denotations (i.e., languages with strong transparency), both autoregressive and masked language models successfully learn to emulate semantic relations between expressions. However, when denotations are changed to be context-dependent with the language otherwise unmodified, this ability degrades. Turning to natural language, our experiments with a specific phenomenon— referential opacity—add to the growing body of evidence that current language models do not represent natural language semantics well. We show this failure relates to the context- dependent nature of natural language form- meaning mappings. 1 Introduction Despite language models’ (LMs) centrality to recent progress on NLP benchmarks, a formal characterization of what can be learned from un- supervised training on large text corpora, and of what modern language models actually do learn, remains elusive. Empirically, Tenney et al. (2019), Kovaleva et al. (2019), Wu et al. (2021), among others, all discovered that pretrained LMs possess unsatisfactory semantic representations. Traylor et al. (2021) found co-variation between form and meaning to be insufficient for an LM to represent lexical semantics. Li et al. (2021), on the other hand, identified evidence of LMs repre- ∗This work was done when Zhaofeng Wu was at AI2. Our code and trained models are released at https:// github.com/ZhaofengWu/transparency. senting dynamic semantics (Kamp, 1981; Heim, 1982; Groenendijk and Stokhof, 1991). From first principles, Bender and Koller (2020) argued that it is a priori impossible for an un- grounded system that has access only to linguistic forms to learn the mapping between those forms and their grounded denotations. They claimed, as a thought experiment, that a learner that has ac- cess to all Java code (i.e., form) on GitHub can never learn execution (i.e., meaning). They nev- ertheless acknowledged that the existence of unit tests, which assert the expected output given input to blocks of code, could constitute a weak form of grounding which potentially enables the learn- ing of meaning. Formalizing this idea, Merrill et al. (2021) the- oretically proved the possibility of learning (or more technically, emulating) semantic relations between expressions in a certain class of formal languages—those that are strongly transpar- ent whose expressions have context-independent denotations—using an assertion oracle, analo- gous to the assertions in unit tests. In addition, with an example, they showed the existence of non-emulatable languages even with an assertion oracle. Yet, the practical implications of these the- oretical results have not been explored. While assertions enable the emulation of strongly trans- parent languages, it is unclear if existing LM archi- tectures …

Content truncated to 500 words