Experimenting with Power Divergences for Language Modeling

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4104–4114, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4104 Experimenting with Power Divergences for Language Modeling Matthieu Labeau Shay B. Cohen Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh {mlabeau,scohen}@inf.ed.ac.uk Abstract Neural language models are usually trained us- ing Maximum-Likelihood Estimation (MLE). The corresponding objective function for MLE is derived from the Kullback-Leibler (KL) di- vergence between the empirical probability distribution representing the data and the para- metric probability distribution output by the model. However, the word frequency discrep- ancies in natural language make performance extremely uneven: while the perplexity is usu- ally very low for frequent words, it is espe- cially difficult to predict rare words. To address that, we experiment with several families (α, β and γ) of power divergences, generalized from the KL divergence, for learn- ing language models with an objective differ- ent than standard MLE. Intuitively, these di- vergences should affect the way the proba- bility mass is spread during learning, notably by prioritizing performance on high or low- frequency words. In addition, we implement and experiment with various sampling-based objectives, where the computation of the out- put layer is only done on a small subset of the vocabulary. They are derived as power generalizations of a softmax approximated via Importance Sam- pling, and Noise Contrastive Estimation, for accelerated learning. Our experiments on the Penn Treebank and Wikitext-2 show that these power divergences can indeed be used to pri- oritize learning on the frequent or rare words, and lead to general performance improve- ments in the case of sampling-based learning. 1 Introduction Language models are an important component in many NLP tasks, where they provide prior knowledge on the language used. They are con- ditional models that aim to predict the next to- ken in a sequence: they can be applied to ba- sic units ranging from individual characters to full words, each approach coming with its own bene- fits and limitations (Merity et al., 2018a). Word- level language models have traditionally been based on n-gram counts, obtaining good perfor- mance with smoothing techniques (Kneser and Ney, 1995; Goodman, 2001). Recently, neural networks have shown strong results in language modeling (Bengio et al., 2001), especially recur- rent neural networks (Mikolov et al., 2011b). As previous approaches, like maximum entropy mod- els (Berger et al., 1996), neural language models are trained via Maximum Likelihood Estimation (MLE). Thus, their training cost grows linearly with the number of words in the vocabulary, of- ten making it prohibitively slow. This motivated a large amount of research work, bringing a variety of solutions (Chen et al., 2016). The large vocabulary sizes encountered in train- ing corpora arguably stem from the fact that the frequency distribution of words in a corpus of natural language follows Zipf’s law (Pow- ers, 1998). This also implies that the discrep- ancy between counts of …

Content truncated to 500 words