A Large-scale Comprehensive Abusiveness Detection Dataset with Multifaceted Labels from Reddit

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 552–561 November 10–11, 2021. ©2021 Association for Computational Linguistics 552 A Large-scale Comprehensive Abusiveness Detection Dataset with Multifaceted Labels from Reddit Hoyun Song∗ Soo Hyun Ryu∗† Huije Lee Jong C. Park‡ School of Computing Korea Advanced Institute of Science and Technology {hysong,shryu,jae4258,park}@nlp.kaist.ac.kr Abstract As users in online communities suffer from se- vere side effects of abusive language, many researchers attempted to detect abusive texts from social media, presenting several datasets for such detection. However, none of them contain both comprehensive labels and contex- tual information, which are essential for thor- oughly detecting all kinds of abusiveness from texts, since datasets with such fine-grained fea- tures demand a significant amount of annota- tions, leading to much increased complexity. In this paper, we propose a Comprehensive Abusiveness Detection Dataset (CADD), col- lected from the English Reddit posts, with mul- tifaceted labels and contexts. Our dataset is annotated hierarchically for an efficient anno- tation through crowdsourcing on a large-scale. We also empirically explore the characteristics of our dataset and provide a detailed analysis for novel insights. The results of our experi- ments with strong pre-trained natural language understanding models on our dataset show that our dataset gives rise to meaningful perfor- mance, assuring its practicality for abusive lan- guage detection. 1 Introduction While enhancing the freedom of expression, online discussion has also brought massive harm inflicted by abusive language. To address this problem, nu- merous studies have attempted to automatically identify abusive expressions in social media (No- bata et al., 2016; Kumar et al., 2018; Wiegand et al., 2018; Zampieri et al., 2019; Pedersen, 2019). How- ever, the definition of abusive language varies to those aspects the researchers have considered im- portant (Nobata et al., 2016; Price et al., 2020). These aspects include lexical profanity (Pedersen, ∗ Equal contribution † Present Address: Department of Psychology, University of Michigan, soohyunr@umich.edu. ‡ Corresponding author 2019; Koufakou et al., 2020) or implicit abuse (Ku- mar et al., 2018; Caselli et al., 2020; Wiegand et al., 2021). Other researchers put their importance on distinguishing targeted attacks from profanity (Po- letto et al., 2017; Zampieri et al., 2019) or identi- fying the targets’ demographic characteristics (Ku- mar et al., 2018; Davidson et al., 2019). Hence, detecting abusive language is quite challenging as it is inherently a high-dimensional phenomenon as to how it is expressed or what the speakers intend to convey. Due to the high-dimensionality, devising a model for identifying the abusiveness1 from social me- dia texts calls for a dataset covering comprehen- sive aspects. However, to the best of our knowl- edge, no study has yet provided a dataset with such fine-grained features for the abusiveness detection. Constructing such a dataset with multifaceted and detailed labels demands a significant amount of annotations, leading to much increased complexity for dataset construction. To address this challenge, we propose a new annotation scheme that exploits the hierarchical structure of interrelated features. Our hierarchical annotation scheme makes use of …

Content truncated to 500 words