DataScience-Polimi at SemEval-2022 Task 8: Stacking Language Models to Predict News Article Similarity

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1229 - 1234 July 14-15, 2022 ©2022 Association for Computational Linguistics DataScience-Polimi at SemEval-2022 Task 8: Stacking Language Models to Predict News Article Similarity Marco Di Giovanni* † Thomas Tasca* ‡ Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Italy *These authors contributed equally to this work. † {marco.digiovanni, marco.brambilla}@polimi.it ‡ thomas.tasca@mail.polimi.it Marco Brambilla† Abstract In this paper, we describe the approach we de- signed to solve SemEval-2022 Task 8: Multi- lingual News Article Similarity. We collect and use exclusively textual features (title, descrip- tion and body) of articles. Our best model is a stacking of 14 Transformer-based Language models fine-tuned on single or multiple fields, using data in the original language or translated to English. It placed fourth on the original leaderboard, sixth on the complete official one and fourth on the English-subset official one. We observe the data collection as our princi- pal source of error due to a relevant fraction of missing or wrong fields. 1 Introduction SemEval-2022 task 8 (Chen et al., 2022) (Multilin- gual News Article Similarity) is a document-level similarity task on news articles data. The goal is to predict whether two multilingual news articles cover the same real-world happening regardless of their writing style, political spin and tone. The task included resources written in 10 different lan- guages: English, German, Spanish, Turkish, Polish, Arabic, French, Chinese, Italian and Russian (the training dataset included news written only in the first seven languages). This task is interesting as we can apply the obtained approaches to cluster news articles and track the similarity of news coverage between different outlets or regions as done in the Agenda Setting project1. Our best model is a simple but effective Stack- ing of a set of Language Models trained on differ- ent combinations of textual features. We fine-tune 14 Language Models, half of them with original multilingual texts and half with texts translated to English. We select three textual fields from the features extracted by our scraper (title, body and description of the news article), and we fine-tune a model for every combination. 1http://www.euagendas.org/ Our model achieved a maximum Pearson corre- lation score of 0.790 and scored 4th on the leader- board that considers the best test result for each team. However, the final ranking, based on a boot- strapping approach across teams’ submissions to es- timate the expected rank, penalizes us to 6th place since it assumes that submissions are an exploration of the hyperparameter/model configuration space of the system. The assumption, released after the end of the competition, does not hold for our team. Our model mainly struggles with missing or wrong data since the training and evaluation datasets were released as links to scrape due to privacy policies. We noticed that we could not col- lect parts of the datasets, and some of the collected data were clearly wrong (e.g. "Get in touch with us. All rights reserved" as the body of an article). The …

Content truncated to 500 words