Diversifying Neural Text Generation with Part-of-Speech Guided Softmax and Sampling

Proceedings of the 29th International Conference on Computational Linguistics, pages 6547–6563 October 12–17, 2022. 6547 Diversifying Neural Text Generation with Part-of-Speech Guided Softmax and Sampling Zhixian Yang and Pengxuan Xu and Xiaojun Wan Wangxuan Institute of Computer Technology, Peking University Center for Data Science, Peking University The MOE Key Laboratory of Computational Linguistics, Peking University yangzhixian@stu.pku.edu.cn xupengxuan@stu.pku.edu.cn wanxiaojun@pku.edu.cn Abstract Neural text generation models are likely to suf- fer from the low-diversity problem. Various decoding strategies and training-based methods have been proposed to promote diversity only by exploiting contextual features, but rarely do they consider incorporating syntactic structure clues. In this work, we propose using linguistic annotation, i.e., part-of-speech (POS), to guide the text generation. In detail, we introduce POS Guided Softmax to explicitly model two poste- rior probabilities: (i) next-POS, and (ii) next- token from the vocabulary of the target POS. A POS Guided Sampling strategy is further pro- posed to address the low-diversity problem by enriching the diversity of POS. Extensive ex- periments and human evaluations show that, compared with existing state-of-the-art meth- ods, our POS Guided Softmax and Sampling (POSG) can generate more diverse text while maintaining comparable quality.1 1 Introduction Maximum likelihood estimation (MLE) is a stan- dard approach to training a neural text generation model, e.g. Transformer (Vaswani et al., 2017), to generate human-like text. However, existing gen- eration systems often suffer from the low-diversity problem (Holtzman et al., 2020; Welleck et al., 2020), which leads to dull and repetitive genera- tions. This problem unavoidably affects the overall generation quality. We conclude that the low-diversity problem is mainly manifested in two aspects: form and content (Fu et al., 2021; Holtzman et al., 2020; Tevet and Berant, 2021). As shown Table 1, the low form diversity can be reflected in repeating some words, using similar lexicon and syntax, and more. The low content diversity can be expressed as a single and dull content with nothing different. 1Our code is available at https: //github.com/FadedCosine/ POS-Guided-Neural-Text-Generation Several feasible fixes have been proposed, such as post-hoc sampling strategies including tempera- ture (Caccia et al., 2020), top-k (Fan et al., 2018), and nucleus sampling (Holtzman et al., 2020). Re- cently, some works suggest that it is the maximiz- ing likelihood itself that should account for the low- diversity problem (Holtzman et al., 2020; Welleck et al., 2020). Holtzman et al. (2020) think that MLE can not adequately capture the rich diversity and ex- pression in human language. Choi et al. (2020) ar- gue that the imbalanced token distribution inherent in natural language even worsens the low-diversity problem. Based on these analysis, many training- based methods have been proposed. Welleck et al. (2020) propose the unlikelihood training to penal- ize repetition with auxiliary losses. Jiang et al. (2019) propose to utilize dynamically scaling losses conditioned on the token frequency in the training phase. Choi et al. (2020) factorize the probability distribution and design an elaborate token cluster algorithm for a balanced training. Though those encouraging progress has been made, we argue that …

Content truncated to 500 words