Team TMA at SemEval-2022 Task 8: Lightweight and Language-Agnostic News Similarity Classifier

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1178 - 1183 July 14-15, 2022 ©2022 Association for Computational Linguistics Team TMA at SemEval-2022 Task 8: Lightweight and Language-Agnostic News Similarity Classifier Nicolas Stefanovitch European Commission - Joint Research Centre nicolas.stefanovitch@ec.europa.eu Abstract We present our contribution to the SemEval 22 Shared Task 8: Multilingual news article similarity. The approach is lightweight and language-agnostic, it is based on the computa- tion of several lexicographic and embedding- based features, and the use of a simple ML ap- proach: random forests. In a notable departure from the task formulation, which is a ranking task, we tackled this task as a classification one. We present a detailed analysis of the behaviour of our system under different settings. 1 Introduction Detecting similar news is a key component of me- dia monitoring, such as the one done by the Europe Media Monitor1, which collects daily about half a million articles in more than 80 languages. This shared task (Chen et al., 2022) allows to study two important challenges that arise in practice: articles can be similar to several extent and in different ways; massive multilingualism requires language agnostic approaches. We report in this paper the result of an experimental approach to tackle this task. We considered computing simple lexicographic and embedding-based features and using simple ML approach for complexity reasons, having in mind that pair-wise comparison of half a million ar- ticles each day is not possible with heavier solution without massive resource cost. There are two im- portant tasks when building such large scale news clustering: determining which pair of articles are worth comparing, and computing the actual similar- ity between pairs of articles. While this shared task deals only with the second approach, the presented system has been designed to tackle also the first one. 1https://emm.newsbrief.eu/ 2 System Description Our system computes several lexicographic- and embedding-based similarity features, which are fed to a standard random forest. We used the follow- ing hyperparameters: 100 trees and a leaf split parameter of 3 data points. On the train set, these parameters avoided extreme overfitting to the data, while not degrading significantly the performances. We used 5-folds cross validation in order to choose the hyper parameters. For each article we consider separately the simi- larities related to 4 fields of the news item: the title, the description, the first sentence and the snippet. The snippet is constituted of the first 4 sentences comprised within the first 512 characters of the text - everything after was ignored, sentences were not truncated. We present here the set of features that have been computed for each of these fields, and for some cross field (title-description, title-first sen- tence, description-first sentences), all features are normalized. Lexicographic measures: we consider two sets of representation: the set of words, and the set of multi-word expressions (MWE). The MWE were computed by splitting the text around stop- words and punctuation marks, essentially similar to RAKE keyword extraction (Piskorski et al., 2021). The …

Content truncated to 500 words