UTMN at SemEval-2020 Task 11: A Kitchen Solution to Automatic Propaganda Detection

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1858–1864 Barcelona, Spain (Online), December 12, 2020. 1858 UTMN at SemEval-2020 Task 11: A Kitchen Solution to Automatic Propaganda Detection Elena Mikhalkova, Nadezhda Ganzherli, Anna Glazkova, Yuliya Bidulya Tyumen State University Tyumen, Russia e.v.mikhalkova@utmn.ru Abstract The article describes a fast solution to propaganda detection at SemEval-2020 Task 11, based on feature adjustment. We use per-token vectorization of features and a simple Logistic Regression classifier to quickly test different hypotheses about our data. We come up with what seems to us the best solution, however, we are unable to align it with the result of the metric suggested by the organizers of the task. We test how our system handles class and feature imbalance by varying the number of samples of two classes (Propaganda and None) in the training set, the size of a context window in which a token is vectorized and combination of vectorization means. The result of our system at SemEval2020 Task 11 is F-score=0.37. 1 Introduction Propaganda is a complex phenomenon that was studied in psychology (Childs, 1936), sociology (Jackall, 1995; Klaehn and Mullen, 2010), theory of communication (Jowett and O’Donnell, 2018; Severin et al., 1997), pedagogy (Hobbs and McGee, 2014; Smith, 1974), history (Taylor, 2013), linguistics (Lukin, 2013), and other sciences. Naturally, the task of automatic propaganda detection has been set and approached in different ways. Proppy, an on-line service, detects propaganda in news articles and clusters them according to an index of propaganda (Barr´on-Cedeno et al., 2019). It daily analyzes emerging texts, identifies events described in them, discards near-duplicates, and, lastly, computes propaganda index on the basis of n-gram features, vocabulary, its richness, style, readability, and NEws LAndscape (NELA) (Horne et al., 2018). In the Propaganda Analysis Project, the propaganda detection is more focused on locating propaganda within a text. At the Hack News Datathon 1 Task 3 was to detect a fragment containing propaganda. However, it was paired with classifying it according to 18 techniques 2, same as the Fragment Level Classification at the shared task of “Fine-Grained Propaganda Detection” (EMNLP2019) (Da San Martino et al., 2019). The competition of Da San Martino et al. (2020) evaluates these two stages separately. Although it is hard to map results of previous competitions on the current one, the former revealed some life hacks that can be of use in the latter. For example, as noted by H.T. Madabushi (2019), extracting fragments of propaganda is similar to the task of Named Entity Recognition, in that they are both span extraction tasks. Their system ProperGander was partially based on the BERT (Devlin et al., 2019) solution for NER. However, probably, the main lesson learned from the first two events was about the necessity of using BERT (or at least a deep-learning neural network) to achieve a state-of-the-art result. In the current article, we proudly present a no-BERT and even no-deep-learning solution for the task of Span Identification. Our choice not to utilize contextualized vectors such as BERT is …

Content truncated to 500 words