Probing Toxic Content in Large Pre-Trained Language Models

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4262–4274 August 1–6, 2021. ©2021 Association for Computational Linguistics 4262 Probing Toxic Content in Large Pre-Trained Language Models Nedjma Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song, Dit-Yan Yeung Department of Computer Science and Engineering The Hong Kong University of Science and Technology nousidhoum@cse.ust.hk, xzhaoar@connect.ust.hk, tfangaa@connect.ust.hk, yqsong@cse.ust.hk, dyyeung@cse.ust.hk Abstract Large pre-trained language models (PTLMs) have been shown to carry biases towards dif- ferent social groups which leads to the repro- duction of stereotypical and toxic content by major NLP systems. We propose a method based on logistic regression classifiers to probe English, French, and Arabic PTLMs and quan- tify the potentially harmful content that they convey with respect to a set of templates. The templates are prompted by a name of a so- cial group followed by a cause-effect relation. We use PTLMs to predict masked tokens at the end of a sentence in order to examine how likely they enable toxicity towards spe- cific communities. We shed the light on how such negative content can be triggered within unrelated and benign contexts based on evi- dence from a large-scale study, then we ex- plain how to take advantage of our methodol- ogy to assess and mitigate the toxicity trans- mitted by PTLMs. 1 Introduction The recent gain in size of pre-trained language mod- els (PTLMs) has had a large impact on state-of-the- art NLP models. Although their efficiency and usefulness in different NLP tasks is incontestable, their shortcomings such as their learning and repro- duction of harmful biases cannot be overlooked and ought to be addressed. Present work on evaluating the sensitivity of language models towards stereo- typical content involves the construction of assess- ment benchmarks (Nadeem et al., 2020; Tay et al., 2020; Gehman et al., 2020) in addition to the study of the potential risks associated with the use and de- ployment of PTLMs (Bender et al., 2021). Previous work on probing PTLMs focuses on their syntac- tic and semantic limitations (Hewitt and Manning, 2019; Marvin and Linzen, 2018), lack of domain- specific knowledge (Jin et al., 2019), and absence of commonsense (Petroni et al., 2019; Lin et al., 2020). However, except for a recent evaluation pro- cess of hurtful sentence completion (Nozza et al., 2021), we notice a lack of large-scale probing ex- periments for quantifying toxic content in PTLMs or systemic methodologies to measure the extent to which they generate harmful content about dif- ferent social groups. In this paper, we present an extensive study which examines the generation of harmful con- tent by PTLMs. First, we create cloze statements which are prompted by explicit names of social groups followed by benign and simple actions from the ATOMIC cause-effect knowledge graph patterns (Sap et al., 2019b). Then, we use a PTLM to predict possible reasons for these ac- tions. We look into how BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), …

Content truncated to 500 words