Robust Conversational Agents against Imperceptible Toxicity Triggers

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2831 - 2847 July 10-15, 2022 ©2022 Association for Computational Linguistics Robust Conversational Agents against Imperceptible Toxicity Triggers Ninareh Mehrabi1, Ahmad Beirami2∗, Fred Morstatter1, Aram Galstyan1 1University of Southern California - Information Sciences Institute 2Meta AI ninarehm@usc.edu, beirami@google.com, {fred,galstyan}@isi.edu Abstract Warning: this paper contains content that may be offensive or upsetting. Recent research in Natural Language Process- ing (NLP) has advanced the development of various toxicity detection models with the in- tention of identifying and mitigating toxic lan- guage from existing systems. Despite the abun- dance of research in this area, less attention has been given to adversarial attacks that force the system to generate toxic language and the defense against them. Existing work to gen- erate such attacks is either based on human- generated attacks which is costly and not scal- able or, in case of automatic attacks, the at- tack vector does not conform to human-like lan- guage, which can be detected using a language model loss. In this work, we propose attacks against conversational agents that are impercep- tible, i.e., they fit the conversation in terms of coherency, relevancy, and fluency, while they are effective and scalable, i.e., they can auto- matically trigger the system into generating toxic language. We then propose a defense mechanism against such attacks which not only mitigates the attack but also attempts to main- tain the conversational flow. Through auto- matic and human evaluations, we show that our defense is effective at avoiding toxic language generation even against imperceptible toxicity triggers while the generated language fits the conversation in terms of coherency and rele- vancy. Lastly, we establish the generalizability of such a defense mechanism on language gen- eration models beyond conversational agents. 1 Introduction Adversarial attacks on different Machine Learn- ing (ML) and Natural Language Processing (NLP) applications can reveal important vulnerability is- sues related to these systems. Most existing re- search focuses on adversarial attacks that degrade performance of existing ML systems with regards ∗ Currently at Google Research. ��������������������������������� ������������� ������������������������������� ��������������������������� �������������������������������� �������������������������� �������������������������������� ������������������ ���������������������������������� ������������� ������� Figure 1: An example illustrating the attack performed by the adversary on the third turn of the conversation (red line) that leads the defender into generating a toxic utterance (dotted box). With a proper defense the de- fender can bypass the attack and generate a non-toxic response (green line). to accuracy (Chakraborty et al., 2018; Zhang et al., 2020b). More recent work has considered attacks that target ethical concerns, such as triggering the models into outputting unfair predictions (Mehrabi et al., 2021b; Solans et al., 2021), or in the context of NLP, generating biased (Sheng et al., 2020) and toxic (Wallace et al., 2019) text. In this paper, we consider adversarial attacks on human-centric chatbots and dialogue systems. It is important for these systems to be safe and robust in the face of natural(-looking) human con- versations. Further, the defender …

Content truncated to 500 words