Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4537–4546, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4537 Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack Emily Dinan Facebook AI Research edinan@fb.com Samuel Humeau Facebook AI Research samuel.humeau@gmail.com Bharath Chintagunta Virginia Tech jaic4@vt.edu Jason Weston Facebook AI Research jase@fb.com Abstract The detection of offensive language in the con- text of a dialogue has become an increasingly important application of natural language pro- cessing. The detection of trolls in public fo- rums (Gal´an-Garc´ıa et al., 2016), and the de- ployment of chatbots in the public domain (Wolf et al., 2017) are two examples that show the necessity of guarding against adversarially offensive behavior on the part of humans. In this work, we develop a training scheme for a model to become robust to such human attacks by an iterative build it, break it, fix it strategy with humans and models in the loop. In de- tailed experiments we show this approach is considerably more robust than previous sys- tems. Further, we show that offensive lan- guage used within a conversation critically de- pends on the dialogue context, and cannot be viewed as a single sentence offensive detec- tion task as in most previous work. Our newly collected tasks and methods are all made open source and publicly available. 1 Introduction The detection of offensive language has become an important topic as the online community has grown, as so too have the number of bad actors (Cheng et al., 2017). Such behavior includes, but is not limited to, trolling in public discussion fo- rums (Herring et al., 2002) and via social media (Silva et al., 2016; Davidson et al., 2017), employ- ing hate speech that expresses prejudice against a particular group, or offensive language specif- ically targeting an individual. Such actions can be motivated to cause harm from which the bad actor derives enjoyment, despite negative conse- quences to others (Bishop, 2014). As such, some bad actors go to great lengths to both avoid detec- tion and to achieve their goals (Shachaf and Hara, 2010). In that context, any attempt to automat- ically detect this behavior can be expected to be adversarially attacked by looking for weaknesses in the detection system, which currently can eas- ily be exploited as shown in (Hosseini et al., 2017; Gr¨ondahl et al., 2018). A further example, rele- vant to the natural langauge processing commu- nity, is the exploitation of weaknesses in machine learning models that generate text, to force them to emit offensive language. Adversarial attacks on the Tay chatbot led to the developers shutting down the system (Wolf et al., 2017). In this work, we study the detection of offen- sive language in dialogue with models that are ro- bust to adversarial attack. We develop an auto- matic approach to the “Build it Break it Fix it” strategy originally adopted for …

Content truncated to 500 words