OversampledML at SemEval-2022 Task 8: When multilingual news similarity met Zero-shot approaches

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1171 - 1177 July 14-15, 2022 ©2022 Association for Computational Linguistics OversampledML at SemEval-2022 Task 8: When multilingual news similarity met Zero-shot approaches Mayank Jobanputra ∗ Lorena Martín Rodríguez ∗ Department of Linguistics, University of Tübingen, Tübingen, Germany {mayank.jobanputra, lorena.martin-rodriguez} @uni-tuebingen.de Abstract We investigate the capabilities of pre-trained models without any fine-tuning, for document-level multilingual news similaritya task of SemEval-2022. We utilize title and news content with appropriate pre-processing techniques. Our system derives 14 differ- ent similarity features using a combination of pre-trained MPNet with well-known statisti- cal methods (i.e. TF-IDF, Word Mover’s dis- tance). We formulate the multilingual news similarity task as a regression task and approxi- mate the overall similarity between two news articles using these features. Our best per- forming system achieved a correlation score of 70.1% and was ranked 20th among the 34 participating teams. In this paper, in ad- dition to a system description, we also pro- vide further analysis of our results and an ab- lation study highlighting the strengths and lim- itations of our features. We make our code publicly available at https://github.com/cicl- iscl/multinewssimilarity. 1 Introduction Assessing semantic similarity between two given content pieces has become one of the important natural language processing (NLP) tasks. This task can help researchers estimate the quality of their models for many other tasks such as: machine trans- lation (MT), summarization, question answering (QA), semantic search, dialog, and conversational systems. Extending semantic similarity task to a cross-lingual setup can extend the evaluation bene- fits to cross-lingual tasks as well. Previous works (Agirre et al., 2016; Cer et al., 2017) focus on sen- tence level cross-lingual semantic similarity. The presented shared task, Chen et al., 2022, proposes a novel problem that focuses on document-level semantic similarity based on news articles. The multilingual news similarity task contains monolingual as well as cross-lingual pairs of news ∗ Both authors contributed equally. reports. This setup enables researchers to test their multilingual models on a document-level semantic similarity task. There can be multiple extensions to this, including the clustering of news articles and tracking similarity of news coverage between different outlets or regions. In this work, we investigate the capabilities of pre-trained multilingual language models (LMs) as well as word-embedding models in this task, with- out fine-tuning them on the task data. Our solution pipeline combines pre-trained MPNet (Song et al., 2020) with well-known statistical methods with well-known statistical methods (i.e. TF-IDF, Word Mover’s distance) to derive the semantic similar- ity features between articles using their title and textual content. We use these similarity features to approximate overall similarity between article pairs. Our system performance is encouraging for this task, albeit with room for improvement. In the following sections, we describe our approach and provide a detailed study of the errors made by the system. We also report the results of an ablation study highlighting strengths and limitations of our derived features. 2 Task Setup 2.1 Dataset The shared task introduced …

Content truncated to 500 words