Comparative Evaluation of Label-Agnostic Selection Bias in Multilingual Hate Speech Datasets

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1–23, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1 Detecting Attackable Sentences in Arguments Yohan Jo1 Seojin Bang1 Emaad Manzoor2 Eduard Hovy1 Chris Reed3 1School of Computer Science, Carnegie Mellon University, USA 2Heinz College of Information Systems and Public Policy, Carnegie Mellon University, USA 3Centre for Argument Technology, University of Dundee, UK {yohanj,seojinb}@cs.cmu.edu, {emaad,hovy}@cmu.edu c.a.reed@dundee.ac.uk Abstract Finding attackable sentences in an argument is the first step toward successful refutation in argumentation. We present a first large-scale analysis of sentence attackability in online ar- guments. We analyze driving reasons for at- tacks in argumentation and identify relevant characteristics of sentences. We demonstrate that a sentence’s attackability is associated with many of these characteristics regarding the sentence’s content, proposition types, and tone, and that an external knowledge source can provide useful information about attacka- bility. Building on these findings, we demon- strate that machine learning models can auto- matically detect attackable sentences in argu- ments, significantly better than several base- lines and comparably well to laypeople.1 1 Introduction Effectively refuting an argument is an important skill in persuasion dialogue, and the first step is to find appropriate points to attack in the argu- ment. Prior work in NLP has studied argument quality (Wachsmuth et al., 2017a; Habernal and Gurevych, 2016a) and counterargument genera- tion (Hua et al., 2019; Wachsmuth et al., 2018). But these studies mainly concern an argument’s overall quality and making counterarguments toward the main claim, without investigating what parts of an argument are attackable for successful persuasion. Nevertheless, attacking specific points of an argu- ment is common and effective; in our data of online discussions, challengers who successfully change the original poster’s view are 1.5 times more likely to quote specific sentences of the argument for at- tacks than unsuccessful challengers (Figure 1). In this paper, we examine how to computationally 1Our data and source code are available at: github. com/yohanjo/emnlp20_arg_attack >A society where everyone is equal seems great to me That's one of the big problems with communism - what is equality? Is everyone equal? [...] >it removes some of the basic faults in society, such as poverty, homelessness, joblessness, as well as touching on moral values such as greed, and envy Yes there are problems within society but this doesn't mean there is a fault with society. [...] >I believe a proper Communist society (I.E. one that is not a dictatorship like Joseph Stalin or Fidel Castro) furthermore, it is unlikely we could ever get a true communist society due to human nature. [...] OP: I believe that Communism is not as bad as everyone says Figure 1: A comment to a post entitled “I believe that Communism is not as bad as everyone says”. It quotes and attacks some sentences in the post (red with “>”) detect attackable sentences in arguments. This at- tackability information would help people make persuasive refutations and strengthen an argument by solidifying potentially attackable points. To examine …

Content truncated to 500 words