A Learning-Exploring Method to Generate Diverse Paraphrases with Multi-Objective Deep Reinforcement Learning

Proceedings of the 28th International Conference on Computational Linguistics, pages 2310–2321 Barcelona, Spain (Online), December 8-13, 2020 2310 A Learning-Exploring Method to Generate Diverse Paraphrases with Multi-Objective Deep Reinforcement Learning Mingtong Liu1 ∗, Erguang Yang1, Deyi Xiong2, Yujie Zhang1 †, Yao Meng3, Changjian Hu3, Jinan Xu1, Yufeng Chen1 1 School of Computer Science and Information Technology && Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University, Beijing, China 2 College of Intelligence and Computing, Tianjin University, Tianjin, China 3 Lenovo Research AI Lab, Beijing, China {mingtongliu, 19112037, yjzhang, jaxu, chenyf}@bjtu.edu.cn, dyxiong@tju.edu.cn, {mengyao1, hucj1}@lenovo.com Abstract Paraphrase generation (PG) is of great importance to many downstream tasks in natural language processing. Diversity is an essential nature to PG for enhancing generalization capability and robustness of downstream applications. Recently, neural sequence-to-sequence (Seq2Seq) mod- els have shown promising results in PG. However, traditional model training for PG focuses on optimizing model prediction against single reference and employs cross-entropy loss, which ob- jective is unable to encourage model to generate diverse paraphrases. In this work, we present a novel approach with multi-objective learning to PG. We propose a learning-exploring method to generate sentences as learning objectives from the learned data distribution, and employ re- inforcement learning to combine these new learning objectives for model training. We first de- sign a sample-based algorithm to explore diverse sentences. Then we introduce several reward functions to evaluate the sampled sentences as learning signals in terms of expressive diversity and semantic fidelity, aiming to generate diverse and high-quality paraphrases. To effectively optimize model performance satisfying different evaluating aspects, we use a GradNorm-based algorithm that automatically balances these training objectives. Experiments and analyses on Quora and Twitter datasets demonstrate that our proposed method not only gains a significant increase in diversity but also improves generation quality over several state-of-the-art baselines. 1 Introduction Paraphrase generation (PG) creates different expressions that share the same meaning (e.g., “how far is Earth from Sun” and “what is the distance between Sun and Earth”). It is a crucial technology in many downstream natural language processing (NLP) applications such as question answering (Dong et al., 2017), machine translation (Zhou et al., 2019), and text summarization (Zhao et al., 2018). Diversity is an essential characteristic of human language, as the meaning of a text can often have multiple different expressions. A good paraphrase generation system is often required to conform to two desired properties (Xu et al., 2018b). The first is diversity, capturing a wide range of linguistic variations. The second is fidelity, preserving semantic meanings while paraphrasing. Therefore, we hope to generate diverse paraphrases while ensuring same meaning, which is important for enhancing generalization capability and robustness of downstream applications (Iyyer et al., 2018). As shown in Table 1, we give some examples. These examples express the same meaning but with different diversities. Most recent state-of-the-art approaches to PG (Prakash et al., 2016; Hasan et al., 2016; Gupta et al., 2018) employ neural sequence-to-sequence (Seq2Seq) models, which mainly uses one given reference for …

Content truncated to 500 words