Civil Rephrases Of Toxic Texts With Self-Supervised Transformers

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1442–1461 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1442 Civil Rephrases Of Toxic Texts With Self-Supervised Transformers L´eo Laugier T´el´ecom Paris, Institut Polytechnique de Paris, France leo.laugier@telecom-paris.fr John Pavlopoulos Department of Computer and System Sciences Stockholm University, Sweden annis@aueb.gr Jeffrey Sorensen Google sorenj@google.com Lucas Dixon Google ldixon@google.com Abstract Platforms that support online commentary, from social networks to news sites, are in- creasingly leveraging machine learning to as- sist their moderation efforts. But this pro- cess does not typically provide feedback to the author that would help them contribute ac- cording to the community guidelines. This is prohibitively time-consuming for human mod- erators to do, and computational approaches are still nascent. This work focuses on mod- els that can help suggest rephrasings of toxic comments in a more civil manner. Inspired by recent progress in unpaired sequence-to- sequence tasks, a self-supervised learning model is introduced, called CAE-T51. CAE- T5 employs a pre-trained text-to-text trans- former, which is fine tuned with a denoising and cyclic auto-encoder loss. Experimenting with the largest toxicity detection dataset to date (Civil Comments) our model generates sentences that are more fluent and better at pre- serving the initial content compared to earlier text style transfer systems which we compare with using several scoring systems and human evaluation. 1 Introduction There are many ways to express our opinions. When we exchange views online, we do not al- ways immediately measure the emotional impact of our message. Even when the opinions expressed are legitimate, well-intentioned and constructive, a poor phrasing may make the conversation go awry (Zhang et al., 2018a). Recently, Natural Language Processing (NLP) research has tackled the prob- lem of abusive language detection by developing accurate classification models that flag toxic (or abusive, offensive, hateful) comments (Davidson 1The code can be found at https://github.com/L eoLaugier/conditional-auto-encoder-text- to-text-transfer-transformer. INPUT OFFENSIVE COMMENT you now have to defend this clown along with his russian corruption. GENERATED CIVIL COMMENT you now have to defend this guy from his russian ties........ INPUT OFFENSIVE COMMENT blaming trudeau and the govern- ment is just stupid. GENERATED CIVIL COMMENT blaming trudeau and the liberal government is just wrong. INPUT OFFENSIVE COMMENT dubya2was a moron. GENERATED CIVIL COMMENT dubya was a republican. Table 1: Examples of offensive sentences from the Civil Comments test set and the more civil rephrasing generated by our model. The third example shows that its strategy may involve shifting the original intent, since “republican” is not a non-offensive synonym of “moron”. et al., 2017; Pavlopoulos et al., 2017; Wulczyn et al., 2017; Gamb¨ack and Sikdar, 2017; Fortuna and Nunes, 2018; Zhang et al., 2018a; Van Hee et al., 2018; Zampieri et al., 2019). The prospect of healthier conversations, nudged by Machine Learning (ML) systems, motivates the development of Natural Language Understanding and Generation (NLU and NLG) models that could later be integrated in a system suggesting alternatives to vituperative comments before they are posted. …

Content truncated to 500 words