Stolen Probability: A Structural Weakness of Neural Language Models

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2191–2197 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2191 Stolen Probability: A Structural Weakness of Neural Language Models David Demeter Northwestern University ddemeter@u.northwestern.edu Gregory Kimmel H. Lee Moffitt Cancer Center gregory.kimmel@moffitt.org Doug Downey Allen Institute for AI dougd@allenai.org Abstract Neural Network Language Models (NNLMs) generate probability distributions by applying a softmax function to a distance metric formed by taking the dot product of a prediction vec- tor with all word vectors in a high-dimensional embedding space. The dot-product distance metric forms part of the inductive bias of NNLMs. Although NNLMs optimize well with this inductive bias, we show that this re- sults in a sub-optimal ordering of the embed- ding space that structurally impoverishes some words at the expense of others when assigning probability. We present numerical, theoretical and empirical analyses showing that words on the interior of the convex hull in the embed- ding space have their probability bounded by the probabilities of the words on the hull. 1 Introduction Neural Network Language Models (NNLMs) have evolved rapidly over the years from simple feed forward nets (Bengio et al., 2003) to include recur- rent connections (Mikolov et al., 2010) and LSTM cells (Zaremba et al., 2014), and most recently transformer architectures (Dai et al., 2019; Radford et al., 2019). This has enabled ever-increasing per- formance on benchmark data sets. However, one thing has remained relatively constant: the softmax of a dot product as the output layer. NNLMs generate probability distributions by applying a softmax function to a distance metric formed by taking the dot product of a prediction vector with all word vectors in a high-dimensional embedding space. We show that the dot product dis- tance metric introduces a limitation that bounds the expressiveness of NNLMs, enabling some words to “steal” probability from other words simply due to their relative placement in the embedding space. We call this limitation the stolen probability effect. While the net impact of this limitation is small in terms of the perplexity measure on which NNLMs are evaluated, we show that the limitation results in significant errors in certain cases. As an example, consider a high probability word sequence like “the United States of America” that ends with a relatively infrequent word such as “America”. Infrequent words are often associated with smaller embedding norms, and may end up in- side the convex hull of the embedding space. As we show, in such a case it is impossible for the NNLM to assign a high probability to the infrequent word that completes the high-probability sequence. Numerical, theoretical and empirical analyses are presented to establish that the stolen probabil- ity effect exists. Experiments with n-gram models, which lack this limitation, are performed to quan- tify the impact of the effect. 2 Background In a NNLM, words wi are represented as vectors xi in a high-dimensional embedding space. Some combination of these vectors xc = {xi}i∈c are used to represent the …

Content truncated to 500 words