An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6012–6018, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6012 An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generation Wanyu Du Department of Computer Science University of Virginia Charlottesville, VA 22904 wd5jq@virginia.edu Yangfeng Ji Department of Computer Science University of Virginia Charlottesville, VA 22904 yangfeng@virginia.edu Abstract Generating paraphrases from given sentences involves decoding words step by step from a large vocabulary. To learn a decoder, su- pervised learning which maximizes the likeli- hood of tokens always suffers from the expo- sure bias. Although both reinforcement learn- ing (RL) and imitation learning (IL) have been widely used to alleviate the bias, the lack of direct comparison leads to only a partial image on their benefits. In this work, we present an empirical study on how RL and IL can help boost the performance of gener- ating paraphrases, with the pointer-generator as a base model 1. Experiments on the bench- mark datasets show that (1) imitation learning is constantly better than reinforcement learn- ing; and (2) the pointer-generator models with imitation learning outperform the state-of-the- art methods with a large margin. 1 Introduction Generating paraphrases is a fundamental research problem that could benefit many other NLP tasks, such as machine translation (Bahdanau et al., 2014), text generation (Radford et al., 2019), document summarization (Chopra et al., 2016), and question answering (McCann et al., 2018). Although various methods have been developed (Zhao et al., 2009; Quirk et al., 2004; Barzi- lay and Lee, 2003), the recent progress on para- phrase generation is mainly from neural net- work modeling (Prakash et al., 2016). Particu- larly, the encoder-decoder framework is widely adopted (Cho et al., 2015), where the encoder takes source sentences as inputs and the decoder generates the corresponding paraphrase for each input sentence. 1The data and code for this work can be ob- tained from: https://github.com/ddddwy/ Reinforce-Paraphrase-Generation In supervised learning, a well-known chal- lenge of generating paraphrases is the exposure bias: the current prediction is conditioned on the ground truth during training but on previous pre- dictions during decoding, which may accumulate and propagate the error when generating the text. To address this challenge, prior work (Li et al., 2018) suggests to utilize the exploration strategy in reinforcement learning (RL). However, train- ing with the RL algorithms is not trivial and often hardly works in practice (Dayan and Niv, 2008). A typical way of using RL in practice is to train the model with supervised learning (Ranzato et al., 2015; Shen et al., 2015; Bahdanau et al., 2016), which leverages the supervision information from training data and alleviate the exposure bias to some extent. In the middle ground between RL and supervised learning, a well-known category is imitation learning (IL) (Daum´e et al., 2009; Ross et al., 2011), which has been used in structured prediction (Bagnell et al., 2007) and other sequen- tial prediction …

Content truncated to 500 words