LSX_team5 at SemEval-2022 Task 8: Multilingual News Article Similarity Assessment based on Word- and Sentence Mover’s Distance

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1190 - 1195 July 14-15, 2022 ©2022 Association for Computational Linguistics LSX_team5 at SemEval-2022 Task 8: Multilingual News Article Similarity Assessment based on Word- and Sentence Mover’s Distance Stefan Heil and Karina Kopp {stefan.heil,karina.sultangaleeva}@stud-mail.uni-wuerzburg.de Konstantin Kobs and Albin Zehe and Andreas Hotho {kobs,zehe,hotho}@informatik.uni-wuerzburg.de University of Würzburg Abstract This paper introduces our submission for the SemEval 2022 Task 8: Multilingual News Ar- ticle Similarity. The task of the competition consisted of the development of a model, capa- ble of determining the similarity between pairs of multilingual news articles. To address this challenge, we evaluated the Word Mover’s Dis- tance in conjunction with word embeddings from ConceptNet Numberbatch and term fre- quencies of WorldLex, as well the Sentence Mover’s Distance based on sentence embed- dings generated by pretrained transformer mod- els of Sentence-BERT. To facilitate the com- parison of multilingual articles with Sentence- BERT models, we deployed a Neural Machine Translation system. All our models achieve sta- ble results in multilingual similarity estimation without learning parameters. 1 Introduction The assessment of similarity between documents is a central challenge in the context of information retrieval. Especially the evaluation of similarities between news articles across different languages opens up opportunities for numerous downstream tasks, such as the analysis of regional differences in news coverage of topics or sentiments towards events and news. Task 8 of the SemEval challenge 2022 (Chen et al., 2022) posed the problem of the assessment of similarity of news articles across a variety of languages and provided a dataset of news article pairs in seven languages, as well as a number of bilingual pairs. Apart from an overall similarity score to be estimated in the challenge, a number of scores for similarity in different categories, such as narrative or entities were provided. For our submission for this task, we evaluated the performance of the Word Mover’s Distance (WMD) and Sentence Mover’s Distance (SMD) in the context of similarity assessment of multilin- gual news articles. We participated in all given lan- guages, comparing both document pairs in the same language, as well as pairs in different languages. Our code is available at https://github. com/StefanJMU/SemEval2022_Task_8. We considered WMD and SMD due to their con- ceptual simplicity and capability to integrate well with resources such as pretrained word embeddings or sentence embeddings, produced by state-of-the- art language models. Additionally, they offer the appeal of being themselves parameter-free, and hence independent of labelled training data. The rest of the paper is organized as follows: We introduce the approaches deployed in the chal- lenge submission, as well as the used resources, supplementing the WMD and SMD. Subsequently, we present the results achieved and conclude with a discussion of the results. 2 System Overview Figure 1 shows a schematic overview of the meth- ods we investigated for our submission. We evaluated two different methods for this task, namely the Word Mover’s Distance (WMD) (Kus- ner et al., 2015) and the Sentence Mover’s distance (SMD) (Clark …

Content truncated to 500 words