Implicit Unlikelihood Training: Improving Neural Text Generation with Reinforcement Learning

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1432–1441 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1432 Implicit Unlikelihood Training: Improving Neural Text Generation with Reinforcement Learning Evgeny Lagutin2,3,4 , Daniil Gavrilov1,2,3 , Pavel Kalaidin3,5∗ 1VK, 2VK Lab, 3Moscow Institute of Physics and Technology, 4Skolkovo Innovation Center, Skolkovo Institute of Science and Technology 5Tinkoff lagutin.em@phystech.edu, daniil.gavrilov@vk.com, p.kalaydin@tinkoff.ai Abstract Likelihood training and maximization-based decoding result in dull and repetitive gen- erated texts even when using powerful lan- guage models (Holtzman et al., 2019). Adding a loss function for regularization was shown to improve text generation output by help- ing avoid unwanted properties, such as con- tradiction or repetition (Li et al., 2020). In this work, we propose fine-tuning a language model by using policy gradient reinforcement learning, directly optimizing for better gen- eration. We apply this approach to mini- mizing repetition in generated text, and show that, when combined with unlikelihood train- ing (Welleck et al., 2020), our method further reduces repetition without impacting the lan- guage model quality. We also evaluate other methods for improving generation at training and decoding time, and compare them using various metrics aimed at control for better text generation output. 1 Introduction Language models have become a subject of close attention in the Natural Language Processing field over the past few years. They are widely used not only for unsupervised pre-training, but also for text generation, such as what is implemented in di- alogue systems (Roller et al., 2020). While there are ongoing efforts to develop non-autoregressive models for language modeling, most current state-of-the-art approaches use the autoregressive method of generating text (i.e., word by word). Holtzman et al. (2019) showed that even powerful trained models with a high likelihood value for test data can output repetitive results. Schmidt (2019) argues that the reason for that is train-test dis- crepancy and lack of generalization when running ∗ Work done while at VK. standard maximum likelihood estimation (MLE) training. Unwanted repetition can be remedied at decod- ing and training time. Decoding methods focus on sampling techniques that generate less repeti- tive or incoherent samples, while other methods aim to improve model training to minimize the ef- fects of degeneration. An effective method for re- ducing language model degeneration is unlikeli- hood training (Welleck et al., 2020), where a reg- ularization term forces the model to reduce the probability of generating a token that has already occurred in a sequence. Li et al. (2020) further explored this idea and showed that adding a loss function for regularization to avoid undesirable sequences improves text generation not only by reducing repetition, but also by decreasing contra- diction. Roller et al. (2020) reported that adding unlikelihood training also improves the human- ness of generated text. In this paper, we propose Implicit Unlikeli- hood Training, a method for regularizing output by fine-tuning a language model with policy gra- dient reinforcement learning to improve genera- tion results. We apply this method …

Content truncated to 500 words