SemEval-2022 Task 8: Multilingual news article similarity

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1094 - 1106 July 14-15, 2022 ©2022 Association for Computational Linguistics SemEval-2022 Task 8: Multilingual news article similarity Xi Chen U. Mass. Amherst xchen4@umass.edu Ali Zeynali U. Mass. Amherst azeynali@umass.edu Chico Q. Camargo University of Exeter f.camargo@exeter.ac.uk Fabian Flöck GESIS f.floeck@gmail.com Devin Gaffney Meedan devin@meedan.com Przemyslaw A. Grabowicz U. Mass. Amherst grabowicz@cs.umass.edu Scott A. Hale University of Oxford and Meedan scott.hale@oii.ox.ac.uk David Jurgens University of Michigan jurgens@umich.edu Mattia Samory GESIS mattia.samory@gesis.org Abstract Thousands of new news articles appear daily in outlets in different languages. Understand- ing which articles refer to the same story can not only improve applications like news aggre- gation but enable cross-linguistic analysis of media consumption and attention. However, assessing the similarity of stories in news arti- cles is challenging due to the different dimen- sions in which a story might vary, e.g., two articles may have substantial textual overlap but describe similar events that happened years apart. To address this challenge, we introduce a new dataset of nearly 10,000 news article pairs spanning 18 language combinations annotated for seven dimensions of similarity as SemEval 2022 Task 8. Here, we present an overview of the task, the best performing submissions, and the frontiers and challenges for measuring multilingual news article similarity. While the participants of this SemEval task contributed very strong models, achieving up to 0.818 cor- relation with gold standard labels across lan- guages, human annotators are capable of reach- ing higher correlations, suggesting space for further progress. 1 Introduction Consider the following question: Given a pair of “hard” news articles,1 are they covering the same news story? Answering this question likely re- quires knowing specific aspects of the events cov- ered: what happened, where and when it hap- pened, who was involved, and why and how it hap- pened (Pan and Kosicki, 1993; Klein and Martínez, 2009; Dijk, 1988). 1“Hard news” is characterized as having a high level of newsworthiness demanding immediate publication (Tuchman, 1972). In our use, we aim to exclude opinion, features, and other forms of journalistic pieces not mainly concerned with covering current events as in Flaxman et al. (2016). Effectively modeling the similarity of news sto- ries holds substantial practical benefits in structur- ing the content of the hundreds of thousands of news articles generated every day.2 Given the vol- ume of articles, an effective measure of news story similarity enables clustering and identification of event coverage in news media (Rupnik et al., 2016; Bisandu et al., 2018). Commercial news aggrega- tion services, as provided by, e.g., Google News or Apple News, perform a similar clustering approach, yet are primarily monolingual and have not been made openly available or extensively researched beyond proprietary solutions. In addition, quantify- ing news article similarity allows the comparison of news outlets in terms of their coverage, under- standing which stories consume much of the media agenda, as well as tracking the diffusion of news stories through a media ecosystem and over time. Being able …

Content truncated to 500 words