Nova-Wang at SemEval-2020 Task 12: OffensEmblert: An Ensemble ofOffensive Language Classifiers

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1587–1597 Barcelona, Spain (Online), December 12, 2020. 1587 Nova-Wang at SemEval-2020 Task 12: OffensEmblert: an Ensemble of Offensive Language Classifiers Susan Wang Nova University IMS, Lisbon, Portugal susanwangds@gmail.com Zita Marinho Priberam Labs Institute for Systems and Robotics, IST zam@priberam.com Abstract This paper presents our contribution to the Offensive Language Classification Task (English SubTask A) of Semeval 2020. We propose different Bert models trained on several offensive language classification and profanity datasets, and combine their output predictions in an ensemble model. We experimented with different ensemble approaches, such as SVMs, Gradient boosting, AdaBoosting and Logistic Regression. We further propose an under-sampling approach of the current SOLID dataset, which removed the most uncertain partitions of the dataset, increasing the recall of the dataset. Our best model, an average ensemble of four different Bert models, achieved 11th place out of 82 participants with a macro F1 score of 0.91344 in the English SubTask A. 1 Introduction As people turn more and more to the online world for their entertainment, social and communication needs, the anonymity offered by the platforms also draw out diverse and sometimes divisive opinions that can lead to abuse, bullying and mental stress. To ensure civility of online discussions while allowing for opposing ideas to be expressed, we need an effective way to detect offensive language in social media. The SemEval 2020 Task 12: “OffensEval 2: Multilingual Offensive Language Identification in Social Media”(Zampieri et al., 2020), introduces a multi-lingual dataset with 5 languages (Arabic, Danish, Greek, Turkish and English). It is a continuation and extension of SemEval 2019 Task 6: “OffensEval: Identifying and Categorizing Offensive Language in Social Media”(Zampieri et al., 2019b). This paper presents the model submitted to the English stream of SubTask A, focusing on the Offensive Language Classification, where for a given tweet we classify it as offensive (OFF) if it contains any form of profanity or targeted offense, either veiled or direct, and non-offensive (NON) otherwise. The official OffensEval 2020 English Training Dataset (Rosenthal et al., 2020) was labelled using a semi-supervised method with several different models. The dataset provides a score which corresponds to an average prediction confidence of being offensive over all the models (µ) along with the uncertainty of the prediction given by their standard deviation (σ). The main challenge of using this dataset concerns with how to make an effective use of this large dataset as the tweets with mid-range average confidence values (µ) often have high variance (σ) as a result are imprecise and difficult to interpret. To overcome this challenge, we created two sub-sampled datasets (A & B), by removing majority of mid-range score µ (uncertain predictions) and with high variance σ from both sets, and by evenly sample from positive and negative values on the second set (set B). We also incorporated additional datasets into our model to balance out the uncertainty of the semi- supervised set. Rather than creating one single combined dataset, we trained separate models for …

Content truncated to 500 words