WueDevils at SemEval-2022 Task 8: Multilingual News Article Similarity via Pair-Wise Sentence Similarity Matrices

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1235 - 1243 July 14-15, 2022 ©2022 Association for Computational Linguistics WueDevils at SemEval-2022 Task 8: Multilingual News Article Similarity via Pair-Wise Sentence Similarity Matrices Dirk Wangsadirdja and Felix Heinickel and Simon Trapp {name}.{surname}@stud-mail.uni-wuerzburg.de Albin Zehe and Konstantin Kobs and Andreas Hotho {surname}@informatik.uni-wuerzburg.de University of Würzburg Abstract We present a system that creates pair-wise co- sine and arccosine sentence similarity matri- ces using multilingual sentence embeddings obtained from pre-trained SBERT and Univer- sal Sentence Encoder models respectively. For each news article sentence, it searches the most similar sentence from the other article and com- putes an average score. Further, a convolu- tional neural network calculates a total similar- ity score for the article pairs on these matrices. Finally, a random forest regressor merges the previous results to a final score that can option- ally be extended with a publishing date score. 1 Introduction The goal of the Multilingual News Article Similar- ity task (Chen et al., 2022) is to check pairs of mul- tilingual news articles against each other in terms of similarity of their information content. The chal- lenge focuses on what is talked about (time, ge- olocation, shared entities), not how the informa- tion is expressed (writing style, emotional tone, etc.), which is essential for applications such as analyzing the news coverage between different re- gions. The participants’ objective is the creation of a model that rates the similarity of article pairs on a 4-point scale from most (1) to least (4) similar and achieves the highest possible Pearson corre- lation score compared to the gold standard. The languages covered by this competition are English (en), German (de), Spanish (es), Turkish (tr), Pol- ish (pl), French (fr) and Arabic (ar) in the training, additionally Italian (it), Russian (ru) and Chinese (zh) in the evaluation. Our system uses an ensemble approach to score pair-wise sentence similarity matrices of the article pairs, which are created with SBERT and Universal Sentence Encoder sentence embeddings. Scores are obtained through simple matrix operations and our convolutional neural network SimCNN based on the TextCNN by Kim (2014). Finally, a ran- dom forest regressor (Breiman, 2001) consolidates the individual scores into a final result, which we extend with a publishing date score. The key challenge of this task is the usage of mul- tilingual text pairs, which requires the application of less precise multilingual language models. Also the splitting of sentences and named entity recog- nition becomes hard to accomplish, since models for these tasks are still monolingual in most cases and we found the few multilingual ones to be un- reliable. The article scraping tool provided by the authors also did not reveal meaningful features to work with besides the article title and text, since the scraped keywords, tags and publishing dates were not always available and in a utilizable format. In the competition, we took 9th place with a Pearson correlation coefficient of 0.759 on the eval- uation set, which differs from …

Content truncated to 500 words