BERT-Beta: A Proactive Probabilistic Approach to Text Moderation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8667–8675 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8667 BERT-β: A Proactive Probabilistic Approach to Text Moderation Fei Tan, Yifan Hu, Kevin Yen, Changwei Hu Yahoo Research, New York, NY, USA {fei.tan, yifanhu, kevinyen, changweih}@yahooinc.com Abstract Text moderation for user generated content, which helps to promote healthy interaction among users, has been widely studied and many machine learning models have been pro- posed. In this work, we explore an alternative perspective by augmenting reactive reviews with proactive forecasting. Specifically, we propose a new concept text toxicity propensity to characterize the extent to which a text tends to attract toxic comments. Beta regression is then introduced to do the probabilistic mod- eling, which is demonstrated to function well in comprehensive experiments. We also pro- pose an explanation method to communicate the model decision clearly. Both propensity scoring and interpretation benefit text moder- ation in a novel manner. Finally, the proposed scaling mechanism for the linear model offers useful insights beyond this work. 1 Introduction Text moderation is essential for maintaining a non- toxic online community for media platforms (No- bata et al., 2016). Many efforts from both academia and industry have been made to address this critical problem. Recently, the most prototypical thread is to do sophisticated feature engineering or develop powerful learning algorithms (Nobata et al., 2016; Badjatiya et al., 2017; Bodapati et al., 2019; Tan et al., 2020; Tran et al., 2020). Automatic com- ment moderation schemes plus human review are certainly the cornerstone of the fight against toxic- ity. These existing works, however, are reactive ap- proaches to handling user generated text in re- sponse to the publication of new articles. In this paper, we revisit this challenge from a proactive perspective. Specifically, we introduce a novel con- cept text toxicity propensity to quantify how likely an article is prone to incur toxic comments. This is a proactive outlook index for news articles prior to the publication, which differs radically from the existing reactive approaches to comments. In this context, reactive describes comment-level moderation algorithms after the publication of news articles (e.g., Perspective (Perspectiveapi)), which quantifies whether comments are toxic and should be taken down or sent for human review. Proac- tive emphasizes article-level moderation effort be- fore the publication (without access to comments), which forecasts how likely articles are to attract toxic comments in the future and gives suggestions (e.g., rephrase news articles properly) in advance. Our work can be viewed as the first machine learn- ing effort for a proactive stance against toxicity. Formally, we propose a probabilistic approach based on Beta distribution (Beta) to regress article toxicity propensity on article text. For previously published news articles with comments, we take the average of comments’ toxicity scores as the ground-truth label for model learning. The effec- tiveness of this approach is shown in both test set and human labeling. We also develop a scheme that can provide convincing explanation to the decision …

Content truncated to 500 words