Probing Neural Network Comprehension of Natural Language Arguments

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658–4664 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4658 Probing Neural Network Comprehension of Natural Language Arguments Timothy Niven and Hung-Yu Kao Intelligent Knowledge Management Lab Department of Computer Science and Information Engineering National Cheng Kung University Tainan, Taiwan tim.niven.public@gmail.com, hykao@mail.ncku.edu.tw Abstract We are surprised to find that BERT’s peak per- formance of 77% on the Argument Reasoning Comprehension Task reaches just three points below the average untrained human baseline. However, we show that this result is entirely accounted for by exploitation of spurious sta- tistical cues in the dataset. We analyze the nature of these cues and demonstrate that a range of models all exploit them. This anal- ysis informs the construction of an adversarial dataset on which all models achieve random accuracy. Our adversarial dataset provides a more robust assessment of argument compre- hension and should be adopted as the standard in future work. 1 Introduction Argumentation mining is the task of determin- ing argumentative structure in natural language text - e.g., which text segments represent claims, and which comprise reasons that support or attack those claims (Mochales and Moens, 2011; Lippi and Torroni, 2016). This is a challenging task for machine learners, as it can be hard even for hu- mans to determine when two text segments stand in argumentative relation, as evidenced by studies on argument annotation (Habernal et al., 2014). One approach to this problem is to focus on warrants (Toulmin, 1958) - a form of world knowledge that permit inferences. Consider a sim- ple argument: “(1) It is raining; therefore (2) you should take an umbrella.”1 The warrant “(3) it is bad to get wet” could license this inference. Knowing (3) facilitates drawing the inferential connection between (1) and (2). However it would be hard to find it stated anywhere since warrants are most often left implicit (Walton, 2005). Thus, on this approach, machine learners must not only reason with warrants but also discover them. 1This example adapted from Black and Hunter (2012) Claim Google is not a harmful monopoly Reason People can choose not to use Google Warrant Other search engines don’t redirect to Google Alternative All other search engines redirect to Google Reason (and since) Warrant → Claim Reason (but since) Alternative → ¬ Claim Figure 1: An example of a data point from the ARCT test set and how it should be read. The inference from R and A to ¬C is by design. The Argument Reasoning Comprehension Task (ARCT) (Habernal et al., 2018a) defers the prob- lem of discovering warrants and focuses on in- ference. An argument is provided, comprising a claim C and reason R. This task is to pick the cor- rect warrant W over a distractor, called the alter- native warrant A. The alternative is written such that R ∧ A → ¬C. An alternative warrant for our earlier example could be “(4) it is good to get …

Content truncated to 500 words