KOLD: Korean Offensive Language Dataset

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10818 - 10833 December 7-11, 2022 ©2022 Association for Computational Linguistics KOLD: Korean Offensive Language Dataset Younghoon Jeong† Juhyun Oh Jongwon Lee Jaimeen Ahn† Jihyung Moon‡ Sungjoon Park‡ Alice Oh† †School of Computing, KAIST ‡SoftlyAI Research, SoftlyAI hoon2j@gmail.com, 411juhyun@snu.ac.kr, mybizzer@gmail.com, jaimeen01@kaist.ac.kr, {jihyung.moon, sungjoon.park}@softly.ai, alice.oh@kaist.edu Abstract Warning: this paper contains content that may be offensive or upsetting. Recent directions for offensive language detec- tion are hierarchical modeling, identifying the type and the target of offensive language, and interpretability with offensive span annotation and prediction. These improvements are focused on English and do not transfer well to other languages be- cause of cultural and linguistic differences. In this paper, we present the Korean Offensive Language Dataset (KOLD) comprising 40,429 comments, which are annotated hierarchically with the type and the target of offensive lan- guage, accompanied by annotations of the cor- responding text spans. We collect the comments from NAVER news and YouTube platform and provide the titles of the articles and videos as the context information for the annotation pro- cess. We use these annotated comments as training data for Korean BERT and RoBERTa models and find that they are effective at offensive- ness detection, target classification, and target span detection while having room for improve- ment for target group classification and offen- sive span detection. We discover that the target group distribution differs drastically from the existing English datasets, and observe that pro- viding the context information improves the model performance in offensiveness detection (+0.3), target classification (+1.5), and target group classification (+13.1). We publicly re- lease the dataset and baseline models. 1 Introduction Online offensive language is a growing societal problem. It propagates negative stereotypes about the targeted social groups, causing representational harm (Barocas et al., 2017). Among various re- search directions for offensive language detection, Level A Feminism is not about gender inequality, It is a mental disorder Feminism is not about gender inequality, It is a mental disorder (KO) GS25, 브레이브걸스 포스터 또 젠더 이슈 논란 (EN) GS25’s poster of Brave Girls evokes dispute over gender issues yet again (KO) 페미는 젠더가 아니라 정신병이다 (EN) Feminism is not about gender inequality, It is a mental disorder Comment Offensive span? Group Individual Others Untargeted Target group? Feminist Offensive? Yes Context (Title) Target span? Level B Level C Target type? Figure 1: An illustration of the annotation process of KOLD. The title is given to provide context to the an- notators. Along with the categorical labels, annotators are asked to mark spans of a sentence that justifies their decision. Waseem et al. (2017) proposes a taxonomy to dis- tinguish hate speech and cyberbullying based on whether the offensive language is directed toward a specific individual or entity, or toward a generalized group. Zampieri et al. (2019) integrates this taxon- omy into a hierarchical annotation process to cre- ate the Offensive Language Identification Dataset (OLID), which is adopted in other languages as well (Zeinert et al., 2021; Zampieri …

Content truncated to 500 words