On the Robustness of Offensive Language Classifiers

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7424 - 7438 May 22-27, 2022 c⃝2022 Association for Computational Linguistics On the Robustness of Offensive Language Classifiers ∗† Jonathan Rusert University of Iowa jonathan-rusert @uiowa.edu Zubair Shafiq University of California, Davis zshafiq@ucdavis.edu Padmini Srinivasan University of Iowa padmini-srinivasan @uiowa.edu Abstract Social media platforms are deploying machine learning based offensive language classifica- tion systems to combat hateful, racist, and other forms of offensive speech at scale. However, despite their real-world deployment, we do not yet comprehensively understand the extent to which offensive language classifiers are ro- bust against adversarial attacks. Prior work in this space is limited to studying robustness of offensive language classifiers against prim- itive attacks such as misspellings and extrane- ous spaces. To address this gap, we system- atically analyze the robustness of state-of-the- art offensive language classifiers against more crafty adversarial attacks that leverage greedy- and attention-based word selection and context- aware embeddings for word replacement. Our results on multiple datasets show that these crafty adversarial attacks can degrade the accu- racy of offensive language classifiers by more than 50% while also being able to preserve the readability and meaning of the modified text. 1 Introduction Online social media platforms are dealing with an unprecedented scale of offensive (e.g., hateful, threatening, profane, racist, and xenophobic) lan- guage (Twitter; Facebook; Reddit). Given the scale of the problem, online social media platforms now increasingly rely on machine learning based sys- tems to proactively and automatically detect offen- sive language (Rosen, 2020; Gadde and Derella, 2020; Kastrenakes, 2019; Hutchinson, 2020). The research community is actively working to im- prove the quality of offensive language classifica- tion (Zampieri et al., 2020, 2019b; Liu et al., 2019; Nikolov and Radivchev, 2019; Mahata et al., 2019; Arango et al., 2020; Agrawal and Awekar, 2018; ∗This paper examines offensive language as a case study. The reader is cautioned that the paper contains unavoidable strong language given the nature of the research. † Our code and data are available at: https://github.com/JonRusert/RobustnessOfOffensiveClassifiers Fortuna and Nunes, 2018). A variety of offensive language classifiers ranging from traditional shal- low models (SVM, Random Forest), deep learn- ing models (CNN, LSTM, GRU), to transformer- based models (BERT, GPT-2) have been proposed in prior literature (Liu et al., 2019; Nikolov and Radivchev, 2019; Mahata et al., 2019). Amongst these approaches, BERT-based transformer models have achieved state-of-the-art performance while ensembles of deep learning models also generally perform well (Zampieri et al., 2019b, 2020). It remains unclear whether the state-of-the-art offensive language classifiers are robust to adversar- ial attacks. While adversarial attacks are of broad interest in the ML/NLP community (Hsieh et al., 2019; Behjati et al., 2019), they are of particular interest for offensive language classification be- cause malicious users can make subtle perturba- tions such that the offensive text is still intelligible to humans but evades detection by machine learn- ing classifiers. Prior work on the robustness of text classification is limited to …

Content truncated to 500 words