Hitachi at SemEval-2020 Task 12: Offensive Language Identification with Noisy Labels Using Statistical Sampling and Post-Processing

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1961–1967 Barcelona, Spain (Online), December 12, 2020. 1961 Hitachi at SemEval-2020 Task 12: Offensive Language Identification with Noisy Labels using Statistical Sampling and Post-Processing Manikandan Ravikiran1∗, Amin Ekant Muljibhai 2∗, Toshinori Miyoshi2 , Hiroaki Ozaki2, Yuta Koreeda2, Masayuki Sakata2 1Research & Development Center, Hitachi India Pvt Ltd., Bangalore, India 2Research & Development Group, Hitachi, Ltd., Tokyo, Japan manikandan@hitachi.co.in,ekant.amin.mu@hitachi.com {toshinori.miyoshi.pd,hiroaki.ozaki.yu}@hitachi.com {yuta.koreeda.pb,masayuki.sakata.nm}@hitachi.com Abstract In this paper, we present our participation in SemEval-2020 Task-12 Subtask-A (English Language) which focuses on offensive language identification from noisy labels. To this end, we developed a hybrid system with the BERT classifier trained with tweets selected using Statistical Sampling Algorithm (SA) and Post-Processed (PP) using an offensive wordlist. Our developed system achieved 34th position with Macro-averaged F1-score (Macro-F1) of 0.90913 over both offensive and non-offensive classes. We further show comprehensive results and error analysis to assist future research in offensive language identification with noisy labels. 1 Introduction Offensive language identification focuses on identifying abusive languages, such as threats, insults, calumniation, discrimination, swearing (Zampieri et al., 2019a; Zampieri et al., 2020) which could be targeted at an individual or group. Typically, offensive language identification is helpful for (i) content moderators who face an increasing volume of abusive content and would like assistance in managing it efficiently (ii) social media sites which do auto-censoring of violence and hate speech and (iii) search query optimization (iv) improving conversations across messenger platforms (Yenala et al., 2017). In recent times offensive language identification has seen a range of works including the creation of a large number of the corpus, open challenges, and benchmarking competitions. Moreover, with recent advances in the area of language understanding and transfer learning in NLP, offensive language identification has seen large traction (Zampieri et al., 2019b). While all these have significantly improved the capacity to identify offensive languages, there are some challenges that require addressing namely (i) Noisy labels as a result of task formulation of SemEval-2020 Task 12 Subtask-A (English Language) which focuses on the identification of offensive language with noisy labels to mitigate the problem of need of large dataset (Rosenthal et al., 2020) and (ii) identification rare and idiosyncratic offensive words (Nobata et al., 2016; Qian et al., 2018). In this paper, we aim to address this task by developing a hybrid system with two major contributions, respectively solving previously mentioned challenges. As a first contribution, we introduce a statistical sampling algorithm that selects trustworthy tweets using statistics of noisy labels and auxiliary datasets. While sampling datasets to reduce noise has been widely explored (Fr´enay and Verleysen, 2014), to the best of our knowledge ours is the very first work in the context of offensive language identification. As a second contribution, we concentrate on the post-processing method that uses words from an offensive wordlist to identify tweets with rare and idiosyncratic offensive words. With these, our proposed system achieved 34th place with Macro-averaged F1-score (Macro-F1) of 0.90913 over both offensive and non-offensive classes. The rest …

Content truncated to 500 words