Evaluating Multilingual Sentence Representation Models in a Real Case Scenario

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2928–2939 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2928 Evaluating Multilingual Sentence Representation Models in a Real Case Scenario Rocco Tripodi1, Rexhina Blloshmi2, Simon Levis Sullam3 1Universisty of Bologna, 2Sapienza University of Rome, 3Ca’ Foscari University of Venice rocco.tripodi@unibo.it, blloshmi@di.uniroma1.it, levissmn@unive.it Abstract In this paper, we present an evaluation of sentence representation models on the paraphrase detection task. The evaluation is designed to simulate a real-world problem of plagiarism and is based on one of the most important cases of forgery in modern history: the so-called “Protocols of the Elders of Zion”. The sentence pairs for the evaluation are taken from the infamous forged text “Protocols of the Elders of Zion” (Protocols) by unknown authors; and by “Dialogue in Hell between Machiavelli and Montesquieu” by Maurice Joly. Scholars have demonstrated that the first text plagiarizes from the second (Cohn, 1967; Taguieff, 2004), indicating all the forged parts on qualitative grounds. Following this evidence, we organized the rephrased texts and asked native speakers to quantify the level of similarity between each pair. We used this material to evaluate sentence representation models in two languages: English and French, and on three tasks: similarity correlation, paraphrase identification, and paraphrase retrieval. Our evaluation aims at encouraging the development of benchmarks based on real-world problems, as a means to prevent problems connected to AI hypes, and to use NLP technologies for social good. Through our evaluation, we are able to confirm that the infamous Protocols are actually a plagiarized text but, as we will show, we encounter several problems connected with the convoluted nature of the task, that is very different from the one reported in standard benchmarks of paraphrase detection and sentence similarity. Code and data available at https://github.com/roccotrip/protocols. Keywords: sentence representation, sentence similarity, paraphrase detection, real-world evaluation 1. Introduction Paraphrase detection is the task of analyzing two seg- ments of text and determining if they have the same meaning despite the differences in structure and word- ing (Wieting et al., 2015). This definition, like that of synonymity for words, is hard to apply in real case sce- narios, since each lexical form has a definite connotation (Bloomfield, 1933). To this end, in this work, we use the denotation by Bhagat and Hovy (2013) of (quasi- ) paraphrases, i.e., sentences or phrases that convey approximately the same meaning using different words. Paraphrase detection is an application of compositional semantics, a discipline that studies how lexical seman- tics units combine to generate complex thoughts, and therefore it is based on the principle of compositionality: the meaning of the whole is a function of the meaning of its parts (Frege, 2002). The first compositional se- mantics models were based on formal semantics (Mon- tague, 2019), while the most recent models are based on distributional semantics (Mitchell and Lapata, 2008; Baroni and Lenci, 2010; Zanzotto et al., 2010; Ji and Eisenstein, 2013) and in particular on neural network …

Content truncated to 500 words