Neural-Driven Search-Based Paraphrase Generation

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2100–2111 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2100 Neural-Driven Search-Based Paraphrase Generation Betty Fabre Orange Labs, Lannion Univ Rennes, IRISA betty.fabre@orange.com Jonathan Chevelu Univ Rennes, IRISA Lannion jonathan.chevelu@irisa.fr Tanguy Urvoy Orange Labs Lannion tanguy.urvoy@orange.com Damien Lolive Univ Rennes, IRISA Lannion damien.lolive@irisa.fr Abstract We study a search-based paraphrase genera- tion scheme where candidate paraphrases are generated by iterated transformations from the original sentence and evaluated in terms of syntax quality, semantic distance, and lexical distance. The semantic distance is derived from BERT, and the lexical quality is based on GPT2 perplexity. To solve this multi-objective search problem, we propose two algorithms: Monte-Carlo Tree Search For Paraphrase Gen- eration (MCPG) and Pareto Tree Search (PTS). We provide an extensive set of experiments on 5 datasets with a rigorous reproduction and validation for several state-of-the-art para- phrase generation algorithms. These experi- ments show that, although being non explic- itly supervised, our algorithms perform well against these baselines. 1 Introduction Paraphrase generation, i.e. the transformation of a sentence into a well-formed but lexically differ- ent one while preserving its original meaning, is a fundamental task of NLP. Its ability to provide diversity and coverage finds applications in several domains like question answering (McKeown, 1979; Harabagiu and Hickl, 2006), machine-translation (Callison-Burch et al., 2006), dialog systems (Yan et al., 2016), privacy (Gr¨ondahl and Asokan, 2019a) or adversarial learning (Iyyer et al., 2018). The formal definition of paraphrase may vary according to the targeted application and the tol- erance we set along several axes, including the semantic distance from the source that we want to minimize, the quality of the syntax, and the lexical distance from the source that we want to maximize to ensure diversity. The available aligned paraphrase corpora are of- ten biased toward specific problems like question answering or image captioning. For instance, a transformer or a seq2seq model trained on a ques- tion answering corpus will typically turn any input sentence into a question. With the lack of generic aligned datasets, it remains challenging to train generic paraphrase models in a supervised manner. On the other hand, with the availability of large- scale non-supervised language models like BERT and GPT2, the assessment of a given candidate paraphrase in terms of semantic distance from its source and lexical quality has become much more tractable. Leveraging from these metrics, we propose to cast the paraphrase generation task as a multi- criteria search problem. We use PPDB 2.0 (Pavlick et al., 2015), a large-scale database of rewriting rules derived from bilingual corpora, to poten- tially generate billions of ’naive’ candidate para- phrases by edition from the source sentence. To sort the good candidates efficiently from the others, we experiment with two search algorithms. The first one, called Monte-Carlo Paraphrase Genera- tion (MCPG), is a variant of the Monte-Carlo Tree Search algorithm (MCTS) (Kocsis and Szepesv´ari, 2006; Gelly and Silver, 2007; Chevelu et al., 2009). The …

Content truncated to 500 words