Paraphrase Generation and Evaluation on Colloquial-Style Sentences

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1814–1822 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1814 Paraphrase Generation and Evaluation on Colloquial-Style Sentences Eetu Sjöblom, Mathias Creutz, Yves Scherrer Department of Digital Humanities, Faculty of Arts, University of Helsinki, Finland {eetu.sjoblom,mathias.creutz,yves.scherrer}@helsinki.fi Abstract In this paper, we investigate paraphrase generation in the colloquial domain. We use state-of-the-art neural machine translation models trained on the Opusparcus corpus to generate paraphrases in six languages: German, English, Finnish, French, Russian, and Swedish. We perform experiments to understand how data selection and filtering for diverse paraphrase pairs affects the generated paraphrases. We compare two different model architectures, an RNN and a Transformer model, and find that the Transformer does not generally outperform the RNN. We also conduct human evaluation on five of the six languages and compare the results to the automatic evaluation metrics BLEU and the recently proposed BERTScore. The results advance our understanding of the trade-offs between the quality and novelty of generated paraphrases, affected by the data selection method. In addition, our comparison of the evaluation methods shows that while BLEU correlates well with human judgments at the corpus level, BERTScore outperforms BLEU in both corpus and sentence-level evaluation. Keywords: paraphrase generation, colloquial language, neural machine translation, evaluation metrics 1. Introduction Paraphrases are a set of sentences or phrases that have the same meaning. The study of paraphrases has both theo- retical and practical implications: On the one hand, it is possible to explore semantic representations that go deeper than surface-level features. Two expressions may carry the same meaning, although they may not contain the same words or their syntactic structures may be completely dif- ferent. These two expressions could have an identical or similar underlying semantic representation or there could be a mapping that transforms one surface form to another. On the other hand, there are practical applications of para- phrase models. Such models can be useful in information retrieval or data mining for discovering expressions with the intended meaning but with totally different surface re- alization than the original query (Riezler et al., 2007). In addition, paraphrasing is used in abstractive summarization as part of summarization models (Nayeem et al., 2018), as well as for evaluation (Vadapalli et al., 2017). Paraphrases can also be used for proofing or grammar checking, produc- ing suggested corrections. Similarly, someone perfecting their skills in a second language, or someone looking for alternate, possibly more idiomatic, expressions may benefit from paraphrase models. For instance, to pick one word, to corroborate, in a few contexts, we can find the follow- ing paraphrase pairs: “She’ll corroborate my story.” → “She’ll back me up.”, “Can you corroborate that?” → “I need proofs.”, “Will people corroborate your account?” → “Is there anybody who can vouch for that?” In this paper, we focus on paraphrase generation using neu- ral machine translation methods. In paraphrase generation we are interested in models that take in an arbitrary in- put sentence …

Content truncated to 500 words