Generative Pretraining for Paraphrase Evaluation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4052 - 4073 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Generative Pretraining for Paraphrase Evaluation Jack Weston, Raphaël Lenain, Udeepa Meepegama, Emil Fristed Novoic {jack, raphael, udeepa, emil}@novoic.com Abstract We introduce ParaBLEU, a paraphrase repre- sentation learning model and evaluation met- ric for text generation. Unlike previous ap- proaches, ParaBLEU learns to understand para- phrasis using generative conditioning as a pre- training objective. ParaBLEU correlates more strongly with human judgements than exist- ing metrics, obtaining new state-of-the-art re- sults on the 2017 WMT Metrics Shared Task. We show that our model is robust to data scarcity, exceeding previous state-of-the-art performance using only 50% of the available training data and surpassing BLEU, ROUGE and METEOR with only 40 labelled examples. Finally, we demonstrate that ParaBLEU can be used to conditionally generate novel para- phrases from a single demonstration, which we use to confirm our hypothesis that it learns ab- stract, generalized paraphrase representations. 1 Introduction Representing the relationship between two pieces of text, be it through a simple algorithm or a deep neural network, has a long history and diverse use- cases that include the evaluation of text generation models (Wiseman et al., 2017; Van Der Lee et al., 2019) and the clinical evaluation of human speech (Johnson et al., 2003; Weintraub et al., 2018). One of the earliest examples of such a representation is the Levenshtein distance (Levenshtein, 1966), which describes the number of character-level edits required to transform one piece of text into an- other. This metric now forms part of a wider fam- ily of edit-distance-based metrics that includes the word error rate (WER) and the translation error rate (TER) (Och, 2003). Other algorithms, such as ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005) and the widely used BLEU metric (Pa- pineni et al., 2002), perform exact or approximate n-gram matching between the two texts. These low-level approaches bear little resem- blance to the human process of comparing two texts, which benefits from a deep prior understand- ing of the semantic and syntactic symmetries of language (Novikova et al., 2017). For example, pairs like “she was no ordinary burglar” and “she was an ordinary burglar” are close in edit-distance- space but semantically disparate. The goal of an automatic text evaluation metric is typically to be a good proxy for human judgements, which is clearly task-dependent. More recently, neural approaches have begun to close the gap between automatic and human judgements of semantic text similarity us- ing Transformer-based language models such as BERT (Zhang et al., 2019a; Sellam et al., 2020). They aim to leverage the transferable knowledge gained by the model during pretraining on large text corpora. The relationship between two texts is similarly modelled, albeit implicitly, by sequence- to-sequence models such as BART (Lewis et al., 2019) and T5 (Raffel et al., 2019). We consider paraphrase evaluation and paraphrase generation to be two instances of paraphrase representation learning. Linguistically, …

Content truncated to 500 words