Principled Paraphrase Generation with Parallel Corpora

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1621 - 1638 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Principled Paraphrase Generation with Parallel Corpora Aitor Ormazabal1, Mikel Artetxe2, Aitor Soroa1, Gorka Labaka1, Eneko Agirre1 1HiTZ Center, University of the Basque Country (UPV/EHU) 2Meta AI {aitor.ormazabal,a.soroa,gorka.labaka,e.agirre}@ehu.eus artetxe@fb.com Abstract Round-trip Machine Translation (MT) is a pop- ular choice for paraphrase generation, which leverages readily available parallel corpora for supervision. In this paper, we formalize the implicit similarity function induced by this ap- proach, and show that it is susceptible to non- paraphrase pairs sharing a single ambiguous translation. Based on these insights, we design an alternative similarity metric that mitigates this issue by requiring the entire translation distribution to match, and implement a relax- ation of it through the Information Bottleneck method. Our approach incorporates an adver- sarial term into MT training in order to learn representations that encode as much informa- tion about the reference translation as possible, while keeping as little information about the input as possible. Paraphrases can be generated by decoding back to the source from this rep- resentation, without having to generate pivot translations. In addition to being more prin- cipled and efficient than round-trip MT, our approach offers an adjustable parameter to con- trol the fidelity-diversity trade-off, and obtains better results in our experiments. 1 Introduction Paraphrase generation aims to generate alternative surface forms expressing the same semantic content as the original text (Madnani and Dorr, 2010), with applications in language understanding and data augmentation (Zhou and Bhat, 2021). One popular approach is to use an MT system to translate the input text into a pivot language and back (Wiet- ing and Gimpel, 2018; Mallinson et al., 2017; Roy and Grangier, 2019). While it intuitively makes sense that translating to another language and back should keep the meaning of a sentence intact while changing its surface form, it is not clear what ex- actly would be considered a paraphrase by such a system. In this work, we show that the probability of a paraphrase xp given a source sentence xs under a en encoder fr decoder en decoder T(xs) representation He will go to school Il va aller à l'école xs y He will attend school ℒadv = –log p(xs | T(xs)) ℒMT = –log p(y | T(xs)) xp inference Figure 1: Proposed system. Given the input xs, we aim to learn a representation T(xs) that encodes as much information as possible about it’s reference translation y (ensuring that the meaning is preserved), and as lit- tle information as possible about xs itself (ensuring that surface information is removed). We achieve this through adversarial learning, where the encoder mini- mizes λLMT − (1 − λ)Ladv and the two decoders min- imize LMT and Ladv. At inference time, we couple the English encoder and decoder to generate a paraphrase xp which, being conditioned on T(x), will preserve the meaning of xs but use a different surface form. …

Content truncated to 500 words