syrapropa at SemEval-2020 Task 11: BERT-based Models Design for Propagandistic Technique and Span Detection

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1808–1816 Barcelona, Spain (Online), December 12, 2020. 1808 syrapropa at SemEval-2020 Task 11: BERT-based Models Design For Propagandistic Technique and Span Detection Jinfen Li College of Arts and Sciences, Syracuse University jli284@syr.edu Lu Xiao School of Information Studies, Syracuse University lxiao04@syr.edu Abstract This paper describes the BERT-based models proposed for two subtasks in SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles. We first build the model for Span Identification (SI) based on SpanBERT, and facilitate the detection by a deeper model and a sentence-level representation. We then develop a hybrid model for the Technique Classification (TC). The hybrid model is composed of three submodels including two BERT models with differ- ent training methods, and a feature-based Logistic Regression model. We endeavor to deal with imbalanced dataset by adjusting cost function. We are in the seventh place in SI subtask (0.4711 of F1-measure), and in the third place in TC subtask (0.6783 of F1-measure) on the development set. 1 Introduction Propaganda exists in social media content and can subvert and distort public deliberation (Farkas, 2018). Natural language processing (NLP) researchers develop computational techniques that automatically detect propaganda in the content. In this paper, we develop two systems for two subtasks in the SemEval-2020 Task 11: (1) Span Identification (SI) (2) Technique Classification (TC) in News Articles respectively. The SI subtask focus on identifying fragments in a given plain-text document which contain at least one propaganda technique; while TC subtask aims to classify the applied propaganda technique given a propagandistic text fragment (Da San Martino et al., 2020). For SI, we interpret the task as to detect a span in a context, which is the smallest detect unit. We base our system on SpanBERT (Joshi et al., 2020), and combines three jointly trained classifiers: sentence, start, and end classifiers. Specifically, the start and end classifiers detect the start and end position of the span, while the sentence classifiers provide sentence-level information for the start and end classifiers. For TC, we come up with a hybrid system combining the two BERT models with different training methods, a Logistic Regression model, and extra rules to classify a propagandistic fragment into one of the 14 techniques. For SI, we find that the segment approach of a context affects the result. While for TC, emotion fea- tures extracted from NRC lexicon are not effective to distinguish 14 classes. However, features such as text length, TF-IDF, occur times in a document, superlative form, question words, hashtags and supple- ment are useful in distinguishing different propaganda techniques. Overall, we are in the seventh place in SI subtask (0.4711 of F1-measure), and in the third place in TC subtask (0.6783 of F1-measure) on the development set. 2 Related Work It is believed that news media plays an active and major role in producing and distributing propaganda and there can be a tool box that helps detect propaganda in news articles (Zollmann, 2019). Da San Martino This work …

Content truncated to 500 words