RuPAWS: A Russian Adversarial Dataset for Paraphrase Identification

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5683–5691 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5683 RuPAWS: A Russian Adversarial Dataset for Paraphrase Identification Nikita Martynov*†, Irina Krotova*†, Varvara Logacheva‡, Alexander Panchenko‡, Olga Kozlova† and Nikita Semenov† †MTS AI, Moscow, Russia ‡Skolkovo Institute of Science and Technology, Moscow, Russia {nimarty1,ivkroto2,oskozlo9,nikita.semenov}@mts.ru, {v.logacheva,a.panchenko}@skoltech.ru Abstract Paraphrase identification task can be easily challenged by changing word order, e.g. as in “Can a good person become bad?”. While for English this problem was tackled by the PAWS dataset (Zhang et al., 2019), datasets for Russian paraphrase detection lack non-paraphrase examples with high lexical overlap. We present RuPAWS, the first adversarial dataset for Russian paraphrase identification. Our dataset consists of examples from PAWS translated to the Russian language and manually annotated by native speakers. We compare it to the largest available dataset for Russian ParaPhraser and show that the best available paraphrase identifiers for the Russian language fail on the RuPAWS dataset. At the same time, the state-of-the-art paraphrasing model RuBERT trained on both RuPAWS and ParaPhraser obtains high performance on the RuPAWS dataset while maintaining its accuracy on the ParaPhraser benchmark. We also show that RuPAWS can measure the sensitivity of models to word order and syntax structure since simple baselines fail even when given RuPAWS training samples. Keywords: paraphrase detection, Russian language, dataset of paraphrases, paraphrasing 1. Introduction Paraphrase identification task can be easily challenged by negative examples with high lexical overlap, e.g. Can a good person become bad? and Can a bad per- son become good? Most of the existing paraphrase identification datasets lack challenging sentence pairs that have a high bag-of-words overlap but are not para- phrases. Negative examples of this type show the sig- nificance of syntax structure and word order. Zhang et al. (2019) emphasize the importance of such ex- amples and introduce PAWS, dataset constructed from Quora and Wikipedia, which consists of adversarial non-paraphrase pairs with high word overlap. Yang et al. (2019) underline the lack of such adversarial ex- amples in existing multilingual datasets for paraphrase identification, e.g. Multi30k (Elliott et al., 2016) and Opusparcus (Creutz, 2018), and create Cross-lingual PAWS (PAWS-X) (Yang et al., 2019). The same challenges exist in the paraphrasing datasets for Russian. The only existing benchmark for Russian sentential paraphrase detection ParaPhraser (Pivo- varova et al., 2017) contains sentence pairs from news headlines and also lacks such challenging examples (See Figure 1). Table 1 shows, that the models trained on ParaPhraser tend to classify the non-paraphrase exam- ples with high BOW overlap as a paraphrase. To bridge this gap and boost the research of Russian paraphrase identification, we create the Russian PAWS (RuPAWS) dataset, the first Russian adversarial dataset for para- phrase classification with high lexical overlap. Our new corpus consists of 8,814 examples from PAWS translated to Russian, cleaned and manually annotated by native speakers. * equal contribution We show that state-of-the-art model for paraphrase identification RuBERT (Kuratov and Arkhipov, 2019) …

Content truncated to 500 words