PAWS: Paraphrase Adversaries from Word Scrambling

Proceedings of NAACL-HLT 2019, pages 1298–1308 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1298 PAWS: Paraphrase Adversaries from Word Scrambling Yuan Zhang Jason Baldridge Luheng He Google AI Language {zhangyua,jridge,luheng}@google.com Abstract Existing paraphrase identification datasets lack sentence pairs that have high lexical over- lap without being paraphrases. Models trained on such data fail to distinguish pairs like flights from New York to Florida and flights from Florida to New York. This paper introduces PAWS (Paraphrase Adversaries from Word Scrambling), a new dataset with 108,463 well- formed paraphrase and non-paraphrase pairs with high lexical overlap. Challenging pairs are generated by controlled word swapping and back translation, followed by fluency and paraphrase judgments by human raters. State- of-the-art models trained on existing datasets have dismal performance on PAWS (<40% accuracy); however, including PAWS train- ing data for these models improves their ac- curacy to 85% while maintaining performance on existing tasks. In contrast, models that do not capture non-local contextual informa- tion fail even with PAWS training examples. As such, PAWS provides an effective instru- ment for driving further progress on models that better exploit structure, context, and pair- wise comparisons. 1 Introduction Word order and syntactic structure have a large im- pact on sentence meaning. Even small perturba- tion in word order can completely change interpre- tation. Consider the following related sentences. (1) Flights from New York to Florida. (2) Flights to Florida from NYC. (3) Flights from Florida to New York. All three have high bag-of-words (BOW) overlap. However, (2) is a paraphrase of (1), while (3) has a very different meaning from (1). Flights from New York to Florida Flights from Florida to New York LM-based Word Scrambling Flights from NYC to Florida Flights from New York to Florida … New York departure flights Flights from Florida to NYC flight from Florida to New York … Looking for flights from Florida +Filtering Backtranslation Flights from New York to Florida Flights from NYC to Florida Flights from Florida to NYC Flights from Florida to New York Positive Positive Negative Recombination PAWS Corpus Original Corpus +Human Judgment +Human Judgment Figure 1: PAWS corpus creation workflow. Existing datasets lack non-paraphrase pairs like (1) and (3). The Quora Question Pairs (QQP) cor- pus contains 400k real world pairs, but its negative examples are drawn primarily from related ques- tions. Few have high word overlap, and of the ∼1,000 pairs with the same BOW, only 20% are not paraphrases. This provides insufficient rep- resentative examples to evaluate models’ perfor- mance on this problem, and there are too few ex- amples for models to learn the importance of word order. Table 1 shows that models trained on QQP are inclined to mark any sentence pairs with high word overlap as paraphrases despite clear clashes in meaning. Models trained or evaluated with only this data may not perform well on real world tasks where such sensitivity is important. To address this, we introduce a workflow (out- lined in Figure 1) …

Content truncated to 500 words