CUE Vectors: Modular Training of Language Models Conditioned on Diverse Contextual Signals

Findings of the Association for Computational Linguistics: ACL 2022, pages 3368 - 3379 May 22-27, 2022 c⃝2022 Association for Computational Linguistics CUE Vectors: Modular Training of Language Models Conditioned on Diverse Contextual Signals Scott Novotney and Sreeparna Mukherjee and Zeeshan Ahmed and Andreas Stolcke Amazon Alexa Seattle, WA, USA {snovotne,sreepar,ahzee,stolcke}@amazon.com Abstract We propose a framework to modularize the training of neural language models that use di- verse forms of sentence-external context (in- cluding metadata) by eliminating the need to jointly train sentence-external and within- sentence encoders. Our approach, contextual universal embeddings (CUE), trains LMs on one set of context, such as date and author, and adapts to novel metadata types, such as article title, or previous sentence. The model consists of a pretrained neural sentence LM, a BERT-based context encoder, and a masked transformer decoder that estimates LM prob- abilities using sentence-internal and sentence- external information. When context or meta- data are unavailable, our model learns to com- bine contextual and sentence-internal informa- tion using noisy oracle unigram embeddings as a proxy. Real contextual information can be introduced later and used to adapt a small num- ber of parameters that map contextual data into the decoder’s embedding space. We validate the CUE framework on a NYTimes text cor- pus with multiple metadata types, for which the LM perplexity can be lowered from 36.6 to 27.4 by conditioning on context. Bootstrap- ping a contextual LM with only a subset of the context/metadata during training retains 85% of the achievable gain. Training the model ini- tially with proxy context retains 67% of the perplexity gain after adapting to real context. Furthermore, we can swap one type of pre- trained sentence LM for another without re- training the context encoders, by only adapting the decoder model. Overall, we obtain a mod- ular framework that allows incremental, scal- able training of context-enhanced LMs. 1 Introduction Language models (LMs) estimate the prior proba- bilities of token sequences and are key probabilistic modeling components in a variety of applications, such as speech recognition, machine translation, or software keyboards. When modeling linguistic token sequences, typical LMs model one sentence or utterance at a time, reflecting the fact that the strongest predictors of words are syntactic con- straints and semantic associations within the sen- tence. However, it has long been recognized that context beyond the sentence has substantial influ- ence on the word probabilities within a sentence. Context literally means the surrounding text (or preceding text, when predicting words in temporal order), but can also refer to any extra-linguistic in- formation, such as metadata (e.g., authorship, time, location) or associated other modalities (e.g., visual cues associated with a spoken utterance). There is a large literature on leveraging such con- textual information for language modeling, some of which we review below (Section 2). However, in- cluding context in language modeling presents ma- jor challenges for operational settings, especially when LMs need to be trained and deployed at scale: • Context data is hard to come by. Many lan- guage …

Content truncated to 500 words