UNT Linguistics at SemEval-2020 Task 12: Linear SVC with Pre-trained Word Embeddings as Document Vectors and Targeted Linguistic Features

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 2209–2215 Barcelona, Spain (Online), December 12, 2020. 2209 UNT Linguistics at SemEval-2020 Task 12: Linear SVC with Pre-trained Word Embeddings as Document Vectors and Targeted Linguistic Features Jared Fromknecht and Alexis Palmer Department of Linguistics, University of North Texas jaredfromknecht@my.unt.edu alexis.palmer@unt.edu Abstract This paper outlines our approach to Tasks A & B for the English Language track of SemEval- 2020 Task 12: OffensEval 2: Multilingual Offensive Language Identification in Social Media. We use a Linear SVM with document vectors computed from pre-trained word embeddings, and we explore the effectiveness of lexical, part of speech, dependency, and named entity (NE) features. We manually annotate a subset of the training data, which we use for error analysis and to tune a threshold for mapping training confidence values to labels. While document vectors are consistently the most informative features for both tasks, testing on the development set suggests that dependency features are an effective addition for Task A, and NE features for Task B. 1 Introduction and System Overview SemEval 2020 Task 12: Offenseval 2 (Zampieri et al., 2020) is an offensive language identification task revolving around classifying social media comments. For the English track, there are three sub-tasks: Offensive Language Identification (Task A), Offense Type Categorization (Task B), and Offense Target Identification (Task C). We focus on Tasks A & B. In contrast to 2019’s task, this year’s training data is labeled using distant supervision: various supervised models trained on the manually-annotated OLID- 2019 data set (Zampieri et al., 2019a) output labels and confidence values for a much larger data set, resulting in the SOLID dataset (Rosenthal et al., 2020). The semi-supervised labels are converted to an average confidence value between 0 and 1; 1 is aligned to the positive class for the task and 0 to the negative class. Thus an additional challenge in this task is determining an appropriate threshold value for mapping confidence measures to labels. To this end, we create a held-out hand-annotated development set from the training data for each task and use it as a test set for picking a good threshold.1 Using Scikit-learn’s Linear SVM implementation (Pedregosa et al., 2011), we build a classifier with features based on two sets of pre-trained word embeddings: GloVe’s 200-dimension Twitter embeddings (Pennington et al., 2014) and 200-dimensional word2vec Twitter embeddings (Deriu et al., 2017). Addi- tionally, we explore four categories of linguistic features beyond the embeddings: lexical, part of speech, dependency, and named entity features. We use spaCy (Honnibal and Johnson, 2015) for preprocessing and extraction of linguistic features. Ablation studies of the additional linguistic features establish which linguistic features are most effective in each task. The goals of these studies are: To further target which features are most informative for each of the sub-tasks; to also narrow in on features that are ineffective or may be detrimental to classification; and to identify which types of offensive constructions are still missed by the model and why. In other …

Content truncated to 500 words