Grounded Compositional Outputs for Adaptive Language Modeling

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1252–1267, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1252 Grounded Compositional Outputs for Adaptive Language Modeling Nikolaos Pappas♣ Phoebe Mulcaire♣ Noah A. Smith♣♦ ♣Paul G. Allen School of Computer Science & Engineering, University of Washington ♦Allen Institute for Artificial Intelligence {npappas,pmulcaire,nasmith@cs.washington.edu} Abstract Language models have emerged as a central component across NLP, and a great deal of progress depends on the ability to cheaply adapt them (e.g., through finetuning) to new domains and tasks. A language model’s vocab- ulary—typically selected before training and permanently fixed later—affects its size and is part of what makes it resistant to such adapta- tion. Prior work has used compositional input embeddings based on surface forms to amelio- rate this issue. In this work, we go one step be- yond and propose a fully compositional output embedding layer for language models, which is further grounded in information from a struc- tured lexicon (WordNet), namely semantically related words and free-text definitions. To our knowledge, the result is the first word-level language model with a size that does not de- pend on the training vocabulary. We evalu- ate the model on conventional language mod- eling as well as challenging cross-domain set- tings with an open vocabulary, finding that it matches or outperforms previous state-of-the- art output embedding methods and adaptation approaches. Our analysis attributes the im- provements to sample efficiency: our model is more accurate for low-frequency words. 1 Introduction Language models (LMs) are at the heart of natu- ral language processing, especially following their recent success in the pretraining paradigm (Dai and Le, 2015; Peters et al., 2018; Devlin et al., 2019; Radford et al., 2019, inter alia). Continued advances in NLP rely on the adaptability of LMs to domains beyond their training data and to new domains and tasks, e.g., through domain adaptive pretraining followed by finetuning (Gururangan et al., 2020). Here, we focus on an important com- ponent of LMs, namely the output vocabulary— over which a LM’s probability distribution over the “next word” (given the history) ranges—and inves- tigate the impact of the type of its representation on the adaptability of neural LMs. Today, LMs are typically trained with a closed output vocabulary derived from the training data; the vocabulary is not modified when the language model is adapted or deployed. This makes large pre- trained language models struggle with rare words, despite being able to produce contextualized rep- resentations for them (Schick and Sch¨utze, 2020). More importantly, this means a generative LM can never give nonzero probability to a specific word it did not see in training. This is a longstanding challenge of language modeling (Jelinek, 1997), but it becomes especially important when we adapt to new domains and tasks. One way to “open up” the vocabulary is to model sequences of bytes, characters, or “word- pieces” rather than the conventional word tokens (Sennrich et al., 2016; Radford et al., 2018; Ponti et al., 2019). While effective, this …

Content truncated to 500 words