Fˆ2-Softmax: Diversifying Neural Text Generation via Frequency Factorized Softmax

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9167–9182, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 9167 F2-Softmax: Diversifying Neural Text Generation via Frequency Factorized Softmax Byung-Ju Choi1 Jimin Hong1 David Keetae Park2 Sang Wan Lee1,3,4 1Humelo, Republic of Korea 2Department of Biomedical Engineering, Columbia University, USA 3Department of Bio and Brain Engineering, KAIST, Republic of Korea 4Center for Neuroscience-inspired AI, KAIST, Republic of Korea {bjej1123, jimin9401}@gmail.com dkp2129@columbia.edu, sangwan@kaist.ac.kr Abstract Despite recent advances in neural text gen- eration, encoding the rich diversity in hu- man language remains elusive. We argue that the sub-optimal text generation is mainly at- tributable to the imbalanced token distribu- tion, which particularly misdirects the learn- ing model when trained with the maximum- likelihood objective. As a simple yet effective remedy, we propose two novel methods, F2- Softmax and MefMax, for a balanced training even with the skewed frequency distribution. MefMax assigns tokens uniquely to frequency classes, trying to group tokens with similar frequencies and equalize frequency mass be- tween the classes. F2-Softmax then decom- poses a probability distribution of the target to- ken into a product of two conditional probabil- ities of (i) frequency class, and (ii) token from the target frequency class. Models learn more uniform probability distributions because they are confined to subsets of vocabularies. Signif- icant performance gains on seven relevant met- rics suggest the supremacy of our approach in improving not only the diversity but also the quality of generated texts. 1 Introduction Neural text generation is one of the extensively studied tasks of natural language processing (NLP), as it forms the basis for dialogue systems (Chen et al., 2017), machine translation (Chaudhary and Patel, 2018), and text summarization (Kryscinski et al., 2019). However, often monotonous or dull, texts generated from existing methods do not fully reflect the rich diversity and expression in human language (Welleck et al., 2020). In particular, mod- els tend to overproduce words frequently appear- ing in the data, while hardly utilizing informa- tive words (Dinan et al., 2020). Even pre-training techniques on large corpora fail to resolve the is- sue (Holtzman et al., 2019). Possible causes for text degeneration have been illuminated, such as a defect specific to model ar- chitectures (Vig, 2018) or the discrepancy between training data and a true distribution (Holtzman et al., 2018; Jiang et al., 2019). Recently, the em- phasis has been placed on investigating the flaws in the maximum likelihood objective (Holtzman et al., 2019). Concretely, the likelihood training pays little attention to the top ranks in terms of the target token probabilities (Welleck et al., 2020), or maximizing likelihood itself does not adequately reflect human language processing (Holtzman et al., 2019). Therefore, with the maximum likelihood- based training, models learn to produce tokens fre- quently appearing in the data more often. We argue, however, that the primary reason be- hind the sub-optimal performance of the likelihood objective is essentially the imbalanced token dis- tribution inherent in natural language. Natural lan- guage is extremely …

Content truncated to 500 words