COLD: A Benchmark for Chinese Offensive Language Detection

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11580 - 11599 December 7-11, 2022 ©2022 Association for Computational Linguistics COLD: A Benchmark for Chinese Offensive Language Detection Disclaimer: The paper contains content that may be profane, vulgar, or offensive. Jiawen Deng1∗ Jingyan Zhou2∗ Hao Sun1 Chujie Zheng1 Fei Mi3 Helen Meng2 Minlie Huang1† 1The CoAI group, DCST, Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems 1Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China 2Dept. of Systems Engineering & Engineering Management, The Chinese University of Hong Kong 3Huawei Noah’s Ark Lab {dengjw2021@mail,aihuang@}tsinghua.edu.cn, jyzhou@se.cuhk.edu.hk Abstract Offensive language detection is increasingly crucial for maintaining a civilized social media platform and deploying pre-trained language models. However, this task in Chinese is still under exploration due to the scarcity of reliable datasets. To this end, we propose a benchmark – COLD for Chinese offensive language anal- ysis, including a Chinese Offensive Language Dataset – COLDATASET and a baseline de- tector – COLDETECTOR which is trained on the dataset. We show that the COLD bench- mark contributes to Chinese offensive language detection which is challenging for existing re- sources. We then deploy the COLDETECTOR and conduct detailed analyses on popular Chi- nese pre-trained language models. We first an- alyze the offensiveness of existing generative models and show that these models inevitably expose varying degrees of offensive issues. Fur- thermore, we investigate the factors that influ- ence the offensive generations, and we find that anti-bias contents and keywords referring to certain groups or revealing negative attitudes trigger offensive outputs easier. 1 1 Introduction Offensive language detection task plays an essen- tial role in maintaining social platforms and pro- moting civilized communication (Davidson et al., 2017; Noever, 2018; Dinan et al., 2019; Jahan and Oussalah, 2021). With the rise of large-scale lan- guage models (Zhang et al., 2020; Roller et al., 2020; Wang et al., 2020; Zhang et al., 2021b), the safety issues due to offensive generation continue to be exposed (Gehman et al., 2020; Xu et al., 2021; Bender et al., 2021; Mi et al., 2022), attracting widespread attention from researchers and push- ing the research boom on this task to new heights ∗The first two authors have equal contribution. †Corresponding author. 1The proposed COLDATASET is released in https://github. com/thu-coai/COLDataset (Sheng et al., 2021; Sun et al., 2021; Dinan et al., 2021). To tackle the problem of offensive language detection, a reliable and versatile benchmark is a needed basis to accelerate in-depth research. The datasets, including WTC (Wulczyn et al., 2017), OLID (Zampieri et al., 2019), BAD (Xu et al., 2020) and RealToxicPrompts (Gehman et al., 2020), are proposed to study the safety issues from different dimensions and granularities. The pub- licly available detector, PerspectiveAPI2, is widely used for toxicity evaluation and contributes to cre- ating safer environments for online communication (Han and Tsvetkov, 2020; Liu et al., 2021). How- ever, most existing works focus on English. The issue of Chinese offensive …

Content truncated to 500 words