Polly Want a Cracker: Analyzing Performance of Parroting on Paraphrase Generation Datasets

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5960–5968, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5960 Polly Want a Cracker: Analyzing Performance of Parroting on Paraphrase Generation Datasets Hong-Ren Mao, Hung-Yi Lee Department of Electrical Engineering National Taiwan University ccm29cam@gmail.com, hungyilee@ntu.edu.tw Abstract Paraphrase generation is an interesting and challenging NLP task which has numerous practical applications. In this paper, we an- alyze datasets commonly used for paraphrase generation research, and show that simply par- roting input sentences surpasses state-of-the- art models in the literature when evaluated on standard metrics. Our findings illustrate that a model could be seemingly adept at gener- ating paraphrases, despite only making trivial changes to the input sentence or even none at all. 1 Introduction The task of paraphrase generation has many im- portant applications in NLP. It can be used to gen- erate adversarial examples of input text, which can then be used to train neural networks so that they become less susceptible to adversarial attack (Iyyer et al., 2018). For knowledge-based QA sys- tems, a paraphrasing step can produce multiple variations of a user query and match them with knowledge base assertions, enhancing recall (Yin et al., 2015; Fader et al., 2014). Relation extraction can also benefit from incorporating paraphrase generation into its processing pipeline (Romano et al., 2006). Manually annotating translation ref- erences is expensive, and automatically generating references through paraphrasing has been shown to be effective for evaluation of machine trans- lation (Zhou et al., 2006; Kauchak and Barzilay, 2006). Datasets used for paraphrase generation in- clude QUORA1, TWITTER (Lan et al., 2017) and MSCOCO (Lin et al., 2014). Previous work on paraphrase generation that used these datasets (Wang et al., 2019; Gupta et al., 2018; Li et al., 1https://data.quora.com/ First-Quora-Dataset-Release-Question-Pairs 2018; Prakash et al., 2016) chose BLEU (Pap- ineni et al., 2002), METEOR (Lavie and Agarwal, 2007) and TER (Snover et al., 2006) as evaluation metrics. In this paper, we find that simply using the input sentence as output in an unsupervised manner (i.e. fully parroting the input) significantly outperforms the state-of-the-art on two metrics for TWITTER, and on one metric for QUORA. Even after chang- ing part of the input sentence (i.e. partially par- roting the input), state-of-the-art metric scores can still be surpassed. Consequently, for future paraphrase generation research which achieve good evaluation scores, we suggest investigating whether their methods or models act differently from simple parroting be- havior. 2 Method Description Given an input sentence i, the goal of paraphrase generation is to generate an output sentence o which is semantically identical to i, but contain variations in lexicon or syntax. Full parroting sim- ply uses the input as output (o = i). Paraphrase generation models may not parrot the input sentence word for word, but it is pos- sible that they only modify a few words of the in- put, thus we also experiment with simple methods …

Content truncated to 500 words