PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3687–3692, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3687 PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification Yinfei Yang∗, Yuan Zhang∗, Chris Tar and Jason Baldridge Google Research, Mountain View, 94043 {yinfeiy, zhangyua, ctar, jasonbaldridge}@google.com Abstract Most existing work on adversarial data gen- eration focuses on English. For example, PAWS (Paraphrase Adversaries from Word Scrambling) (Zhang et al., 2019) consists of challenging English paraphrase identification pairs from Wikipedia and Quora. We rem- edy this gap with PAWS-X, a new dataset of 23,659 human translated PAWS evaluation pairs in six typologically distinct languages: French, Spanish, German, Chinese, Japanese, and Korean. We provide baseline numbers for three models with different capacity to cap- ture non-local context and sentence structure, and using different multilingual training and evaluation regimes. Multilingual BERT (De- vlin et al., 2019) fine-tuned on PAWS En- glish plus machine-translated data performs the best, with a range of 83.1-90.8 accuracy across the non-English languages and an av- erage accuracy gain of 23% over the next best model. PAWS-X shows the effectiveness of deep, multilingual pre-training while also leaving considerable headroom as a new chal- lenge to drive multilingual research that better captures structure and contextual information. 1 Introduction Adversarial examples have effectively highlighted the deficiencies of state-of-the-art models for many natural language processing tasks, e.g. ques- tion answering (Jia and Liang, 2017; Chen et al., 2018; Ribeiro et al., 2018), textual entailment (Zhao et al., 2018; Glockner et al., 2018), and text classification (Alzantot et al., 2018; Iyyer et al., 2018). Zhang et al. (2019) introduce PAWS, which has adversarial paraphrase identification pairs with high lexical overlap, like flights from New York to Florida and flights from Florida to New York. Such pairs stress the importance of modeling sen- tence structure and context because they have high ∗equal contribution word overlap ratio but different semantic meaning. In addition to revealing failures of state-of-the-art models, research on adversarial examples has gen- erally shown that augmenting training data with good adversarial examples can boost performance for some models—providing greater clarity to the modeling landscape as well providing new head- room for further improvements. Most previous work focuses only on English de- spite the fact that the problems highlighted by ad- versarial examples are shared by other languages. Existing multilingual datasets for paraphrase iden- tification, e.g. Multi30k (Elliott et al., 2016) and Opusparcus (Creutz, 2018), lack challenging ex- amples like PAWS. The lack of high-quality ad- versarial examples in other languages makes it difficult to benchmark model improvements. We bridge this gap by creating Cross-lingual PAWS (PAWS-X), an extension of the Wikipedia por- tion of the PAWS evaluation and test examples to six languages: Spanish, French, German, Chi- nese, Japanese, and Korean. This new corpus con- sists of 23,659 human translated example pairs with paraphrase judgments in each target lan- guage. Like previous work on multilingual cor- …

Content truncated to 500 words