Distributionally Robust Language Modeling

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4227–4237, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4227 Distributionally Robust Language Modeling Yonatan Oren*1 Shiori Sagawa*1 Tatsunori B. Hashimoto*1,2 Percy Liang1 (* equal contribution) 1Stanford Computer Science 2Stanford Statistics {yonatano,thashim}@stanford.edu {ssagawa,pliang}@cs.stanford.edu Abstract Language models are generally trained on data spanning a wide range of topics (e.g., news, reviews, fiction), but they might be applied to an a priori unknown target distribution (e.g., restaurant reviews). In this paper, we first show that training on text outside the test dis- tribution can degrade test performance when using standard maximum likelihood (MLE) training. To remedy this without the knowl- edge of the test distribution, we propose an ap- proach which trains a model that performs well over a wide range of potential test distribu- tions. In particular, we derive a new distribu- tionally robust optimization (DRO) procedure which minimizes the loss of the model over the worst-case mixture of topics with sufficient overlap with the training distribution. Our ap- proach, called topic conditional value at risk (topic CVaR), obtains a 5.5 point perplexity re- duction over MLE when the language models are trained on a mixture of Yelp reviews and news and tested only on reviews. 1 Introduction Large-scale language modeling plays a central role in both text generation (Sordoni et al., 2015; Nal- lapati et al., 2016) and unsupervised pre-training (Vaswani et al., 2013; Dai and Le, 2015; McCann et al., 2017; Peters et al., 2018; Devlin et al., 2018; Radford et al., 2018). In both settings, a sin- gle language model is trained on a large corpus containing a range of topics (e.g. news, fiction, and reviews). This language model is then ap- plied in many different tasks, each with a specific test distribution (e.g., analyzing the sentiment of restaurant reviews). Can we train a single general- purpose language model that works across a wide range of potential test distributions? In this work, we first demonstrate that stan- dard maximum likelihood training on a large, het- erogeneous dataset can fail to achieve this goal. x ptrain x (x) reviews news training p x p (x) MLE robust Figure 1. Illustration of a training corpus as a density (black) with mostly news stories (red) and a small number of restaurant reviews (blue). The standard MLE model (gray) reflects the underlying data and assigns little weight to reviews, and thus performs poorly on reviews. A more robust model should try to equalize the weight across all topics so that it can perform well regardless of which topics appear at test time. While more data is generally better, the presence of text outside the target distribution actually de- grades performance on a target test distribution. For example, a language model trained on Yelp reviews achieves a perplexity of 32, and this per- plexity increases to 43 when trained on a mix- ture of 10% Yelp and …

Content truncated to 500 words