Keep the Primary, Rewrite the Secondary: A Two-Stage Approach for Paraphrase Generation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 560–569 August 1–6, 2021. ©2021 Association for Computational Linguistics 560 Keep the Primary, Rewrite the Secondary: A Two-Stage Approach for Paraphrase Generation Yixuan Su♠ David Vandyke♣ Simon Baker♠ Yan Wang♦ Nigel Collier♠ ♠Language Technology Lab, University of Cambridge ♣Apple ♦Tencent AI Lab {ys484,sb895,nhc30}@cam.ac.uk dvandyke@apple.com, brandenwang@tencent.com Abstract Paraphrase generation is an and challenging NLG problem.important In this work, we propose a new Identification-then- Aggregation (IA) framework to tackle this task. In the identification step, the input tokens are sorted into two groups by a novel Primary/Secondary Identification (PSI) algorithm. In the aggregation step, these groups are separately encoded, before being aggregated by a custom designed decoder, which autoregressively generates the para- phrased sentence. In extensive experiments on two benchmark datasets, we demonstrate that our model outperforms previous studies by a notable margin. We also show that the proposed approach can generate paraphrases in an interpretable and controllable way. 1 Introduction Paraphrases refer to text (often sentences) that share the same meaning but use different choices of words and their ordering. Automatic generation of paraphrases is a longstanding problem that is impor- tant to many downstream NLP applications such as question answering (Dong et al., 2017; Buck et al., 2018), machine translation (Cho et al., 2014), and semantic parsing (Su and Yan, 2017). Most early research adopts the sequence-to-sequence model (Prakash et al., 2016; Cao et al., 2017; Li et al., 2018) to map the input text to its paraphrase by processing and generating each word in a uniform way. Rather than processing each word uniformly, some recent studies tackles this task in a decompos- able manner. For instance, Li et al. (2019) adopt an external word aligner to extract paraphrasing patterns at different levels of granularity and then perform generation. Fu et al. (2019) first use source words to predict their neighbors and then organize the predicted neighbors into a complete sentence. Figure 1: Examples of paraphrase pair sampled from Quora and MSCOCO datasets in which the words in red refer to the primary content and the rest of the words make up the secondary content. In this work, we investigate decomposable para- phrase generation from a different perspective. Specifically, we consider using a non-parametric approach to label each token in an input sentence as either (i) primary, or (ii) secondary. Intuitively, the primary content of a sentence refers to the factual information that defines the shared meaning of the paraphrase pair. All other content is deemed as sec- ondary, and typically controls the structure of the sentence. In practice, this distinction is determined by an algorithm that decides whether tokens are primary or secondary, as described in §3. To better illustrate our idea, in Figure 1, we show some exam- ples sampled from Quora and MSCOCO (Lin et al., 2014) datasets. We see that, for many cases, the paraphrase pairs maintain the similar primary con- tent (e.g., the phrases “baby elephants” and “baby elephant” in the first example) while the …

Content truncated to 500 words