Profanity-Avoiding Training Framework for Seq2seq Models with Certified Robustness

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5151–5161 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5151 Profanity-Avoiding Training Framework for Seq2seq Models with Certified Robustness Hengtong Zhang1,3, Tianhang Zheng2, Yaliang Li4, Jing Gao1, Lu Su1 and Bo Li5 1Purdue University, 2Toronto University, 3University at Buffalo, 4Alibaba Group, 5University of Illinois at Urbana-Champaign htzhang.work@gmail.com, th.zheng@mail.utoronto.ca, yaliang.li@alibaba-inc.com, {jinggao, lusu}@purdue.edu, lbo@illinois.edu Abstract Seq2seq models have demonstrated their in- credible effectiveness in a large variety of applications. However, recent research has shown that inappropriate language in train- ing samples and well-designed testing cases can induce seq2seq models to output profanity. These outputs may potentially hurt the usabil- ity of seq2seq models and make the end-users feel offended. To address this problem, we propose a training framework with certified ro- bustness to eliminate the causes that trigger the generation of profanity. The proposed train- ing framework leverages merely a short list of profanity examples to prevent seq2seq models from generating a broader spectrum of profan- ity. The framework is composed of a pattern- eliminating training component to suppress the impact of language patterns with profanity in the training set, and a trigger-resisting train- ing component to provide certified robustness for seq2seq models against intentionally in- jected profanity-triggering expressions in test samples. In the experiments, we consider two representative NLP tasks that seq2seq can be applied to, i.e., style transfer and dialogue gen- eration. Extensive experimental results show that the proposed training framework can suc- cessfully prevent the NLP models from gener- ating profanity. 1 Introductions In the past decade, the research community has wit- nessed machine learning models achieving impres- sive performances in various sequence-to-sequence (Seq2seq) NLP tasks such as style transfer and dialogue generation. Despite their great success, recent studies and reports have shown that widely used models trained on crowd-sourced corpus like user reviews and online community discussions may produce inappropriate languages such as pro- fanity. Such inappropriate languages may hurt the usability of these models and cause conflicts and anxiety among the users. For instance, in 2016, Microsoft released an AI chatbot named Tay1, which is claimed to be able to improve itself through communicating with social media users. Nevertheless, within just 24 hours after Microsoft released the chatbot, it started to generate misogynistic and racist words. Microsoft had to suspend the chatbot account and conceded that the chatbot suffered from a “coordinated attack by a subset of people”. Such an incident demon- strates the vulnerability of existing Seq2seq meth- ods facing users’ abuse. There are two major causes that can make the Seq2seq model produce profanity. First, in the training phase, Seq2seq models can capture the language patterns within the training corpus. Sim- ilarly, languages patterns with profanity (referred to as profanity patterns) in the training corpus also can be learned and concealed in the learned model. Second, in the testing phase, some specific tokens may trigger expressions that contain profanity so that the Seq2seq model can generate inappropriate languages. Sometimes, such triggering expressions …

Content truncated to 500 words