HABERTOR: An Efficient and Effective Deep Hatespeech Detector

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7486–7502, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7486 HABERTOR: An Efficient and Effective Deep Hatespeech Detector Thanh Tran⋆, Yifan Hu♦, Changwei Hu♦, Kevin Yen♦, Fei Tan♦, Kyumin Lee⋆, Serim Park♥ ⋆Worcester Polytechnic Institute, ♦Yahoo! Research, ♥Twitter {tdtran,kmlee}@wpi.edu {yifanhu,changweih,kevinyen,fei.tan}@verizonmedia.com serimp@twitter.com Abstract We present our HABERTOR model for detect- ing hatespeech in large scale user-generated content. Inspired by the recent success of the BERT model, we propose several modifi- cations to BERT to enhance the performance on the downstream hatespeech classification task. HABERTOR inherits BERT’s architec- ture, but is different in four aspects: (i) it gen- erates its own vocabularies and is pre-trained from the scratch using the largest scale hate- speech dataset; (ii) it consists of Quaternion- based factorized components, resulting in a much smaller number of parameters, faster training and inferencing, as well as less mem- ory usage; (iii) it uses our proposed multi- source ensemble heads with a pooling layer for separate input sources, to further enhance its effectiveness; and (iv) it uses a regularized adversarial training with our proposed fine- grained and adaptive noise magnitude to en- hance its robustness. Through experiments on the large-scale real-world hatespeech dataset with 1.4M annotated comments, we show that HABERTOR works better than 15 state-of- the-art hatespeech detection methods, includ- ing fine-tuning Language Models. In particu- lar, comparing with BERT, our HABERTOR is 4∼5 times faster in the training/inferencing phase, uses less than 1/3 of the memory, and has better performance, even though we pre- train it by using less than 1% of the number of words. Our generalizability analysis shows that HABERTOR transfers well to other un- seen hatespeech datasets and is a more effi- cient and effective alternative to BERT for the hatespeech classification. 1 Introduction The occurrence of hatespeech has been increas- ing (Barna, 2019). It has become easier than before to reach a large audience quickly via social media, causing an increase of the temptation for inappro- priate behaviors such as hatespeech, and potential damage to social systems. In particular, hatespeech interferes with civil discourse and turns good peo- ple away. Furthermore, hatespeech in the virtual world can lead to physical violence against cer- tain groups in the real world12, so it should not be ignored on the ground of freedom of speech. To detect hatespeech, researchers developed human-crafted feature-based classifiers (Chatza- kou et al., 2017; Davidson et al., 2017; Waseem and Hovy, 2016; MacAvaney et al., 2019), and pro- posed deep neural network architectures (Zampieri et al., 2019; Gamb¨ack and Sikdar, 2017; Park and Fung, 2017; Badjatiya et al., 2017; Agrawal and Awekar, 2018). However, they might not explore all possible important features for hatespeech de- tection, ignored pre-trained language model un- derstanding, or proposed uni-directional language models by reading from left to right or right to left. Recently, the BERT (Bidirectional Encoder Rep- resentations from Transformers) model (Devlin et al., 2019) has achieved tremendous success in Natural Language Processing . The …

Content truncated to 500 words