Negative Training for Neural Dialogue Response Generation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2044–2058 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2044 Negative Training for Neural Dialogue Response Generation Tianxing He and James Glass Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology {tianxing,glass}@csail.mit.edu Abstract Although deep learning models have brought tremendous advancements to the field of open- domain dialogue response generation, recent research results have revealed that the trained models have undesirable generation behaviors, such as malicious responses and generic (bor- ing) responses. In this work, we propose a framework named “Negative Training” to min- imize such behaviors. Given a trained model, the framework will first find generated samples that exhibit the undesirable behavior, and then use them to feed negative training signals for fine-tuning the model. Our experiments show that negative training can significantly reduce the hit rate of malicious responses, or discour- age frequent responses and improve response diversity. 1 Introduction End-to-end dialogue response generation can be formulated as a sequence-to-sequence (seq2seq) task: given a dialogue context, the model is asked to generate a high-quality response. In recent years, deep learning models, especially seq2seq language generation models (Sutskever et al., 2014; Cho et al., 2014), have brought significant progress to the field of dialogue response generation. However, recent research has revealed undesir- able behaviors of seq2seq models that are side ef- fects of standard maximum likelihood estimation (MLE) training, such as the generic (boring) re- sponse problem (Li et al., 2016), vulnerability to adversarial attacks (Cheng et al., 2018; Belinkov and Bisk, 2017), and the malicious (egregious) re- sponse problem (He and Glass, 2019). In this work, we propose and explore the nega- tive training framework to correct unwanted behav- iors of a dialogue response generator. During nega- tive training, we first find or identify input-output pairs for a trained seq2seq model that exhibit some undesirable generation behavior, treat them as “bad examples,” and use them to feed negative training signals to the model. Correspondingly, we regard the training data as “good examples” and standard MLE training as “positive training”. The idea of negative training is inspired from the way parents might teach their children to use lan- guage by incorporating both positive and negative training signals. For example, when teaching chil- dren how to use “love” and “hate”, in addition to using positive examples like “I love apples but I hate bananas”, they might also point out that saying “I hate you” to someone is con- sidered impolite. In this work, negative training is used to address the malicious response problem and the frequent re- sponse problem (to be described in Section 3.2 and 3.3) in open-domain dialogue response generation. In our experiments, we show that negative training can significantly reduce the hit rate for malicious responses, or discourage frequent responses and greatly improve response diversity. 2 Model Formulation In this work we adopt recurrent neural network (RNN) based encoder-decoder seq2seq models (Sutskever et al., 2014; Cho et al., 2014; Mikolov et …

Content truncated to 500 words