Revisiting Representation Degeneration Problem in Language Modeling

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 518–527 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 518 Revisiting Representation Degeneration Problem in Language Modeling Zhong Zhang1, Chongming Gao1, Cong Xu1, Rui Miao2, Qinli Yang1, Junming Shao1∗ 1University of Electronic Science and Technology of China, Chengdu, China 2Guizhou University, Guiyang, China {zhongzhang,congxu}@std.uestc.edu.cn, chongming.gao@gmail.com, miaorui93@163.com, {qinli.yang,junmshao}@uestc.edu.cn Abstract Weight tying is now a common setting in many language generation tasks such as language modeling and machine translation. However, a recent study reveals that there is a potential flaw in weight tying. They find that the learned word embeddings are likely to degenerate and lie in a narrow cone when training a language model. They call it the representation degen- eration problem and propose a cosine regular- ization to solve it. Nevertheless, we prove that the cosine regularization is insufficient to solve the problem, as the degeneration is still likely to happen under certain conditions. In this pa- per, we revisit the representation degeneration problem and theoretically analyze the limita- tions of the previously proposed solution. Af- terward, we propose an alternative regulariza- tion method called Laplacian regularization to tackle the problem. Experiments on language modeling demonstrate the effectiveness of the proposed Laplacian regularization. 1 Introduction Language modeling is a fundamental task in natural language processing, applications include machine translation (Bahdanau et al., 2015; Vaswani et al., 2017), image captioning (Vinyals et al., 2015; Xu et al., 2015) and speech recognition (Yu and Deng, 2016), to name a few. In the era of deep learning, a general model architecture usually contains a word embedding layer as input, multiple layers to encode word context as a fixed-size hidden state, and a softmax layer to transform the hidden-state into a categorical distribution of the next word (Merity et al., 2018; Yang et al., 2018; Gong et al., 2018; Wang et al., 2019; Gao et al., 2019). While in practice, the parameters of the embedding layer and the softmax layer are usually shared, which is called weight tying (Inan et al., 2017; Press and Wolf, 2017). ∗Corresponding author http://dm.uestc.edu.cn (a) Vanilla (b) CosReg Figure 1: Illustration of the degeneration phenomenon from (Gao et al., 2019). (a). Word embeddings trained from vanilla Transformer. (b). Word embeddings trained with cosine regularization. Despite the improvements from weight tying, a recent work (Gao et al., 2019) discovers that, with weight tying, the learned word embeddings are pos- itively correlated and spread in a narrow cone as visualized in Figure 1(a). A similar phenomenon is observed in Gong et al. (2018). Thus, the seman- tic expressiveness of word embeddings is limited. They call it the representation degeneration prob- lem. To tackle the problem, the authors propose a cosine regularization that minimizes the cosine similarities between any two word embeddings to enlarge the aperture of the cone. They show that it improves the language modeling performance and eases the degeneration as visualized in Figure 1(b). However, we argue that the cosine regularization might not be the best choice …

Content truncated to 500 words