SemEval-2018 Task 12: The Argument Reasoning Comprehension Task

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 763–772 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics SemEval-2018 Task 12: The Argument Reasoning Comprehension Task Ivan Habernal† Henning Wachsmuth‡ Iryna Gurevych† Benno Stein‡ † Ubiquitous Knowledge Processing Lab (UKP) and Research Training Group AIPHES Department of Computer Science, Technische Universit¨at Darmstadt, Germany www.ukp.tu-darmstadt.de www.aiphes.tu-darmstadt.de ‡ Faculty of Media, Bauhaus-Universit¨at Weimar, Germany <firstname>.<lastname>@uni-weimar.de Abstract A natural language argument is composed of a claim as well as reasons given as premises for the claim. The warrant explaining the reason- ing is usually left implicit, as it is clear from the context and common sense. This makes a comprehension of arguments easy for humans but hard for machines. This paper summarizes the first shared task on argument reasoning comprehension. Given a premise and a claim along with some topic information, the goal is to automatically identify the correct war- rant among two candidates that are plausible and lexically close, but in fact imply opposite claims. We describe the dataset with 1970 in- stances that we built for the task, and we out- line the 21 computational approaches that par- ticipated, most of which used neural networks. The results reveal the complexity of the task, with many approaches hardly improving over the random accuracy of ≈ 0.5. Still, the best observed accuracy (0.712) underlines the prin- ciple feasibility of identifying warrants. Our analysis indicates that an inclusion of external knowledge is key to reasoning comprehension. 1 Introduction When we argue in natural language, we give rea- sons as premises for our claims. A fundamental pragmatic instrument in this regard is to leave those parts of an argument unstated that can be presup- posed. This is particularly common for the reason- ing between an argument’s premises and its claim, called implicit warrants there (Toulmin, 1958). A warrant takes the role of an inference rule, i.e., the abstract structure of an argument is reason → (since) warrant → (therefore) claim. In princi- ple, this structure applies to deductive arguments, which allows us to validate arguments properly formalized in propositional logic. However, most natural language arguments are in fact inductive (Govier, 2010) or defeasible (Walton, 2007). Topic: Tax Break for Sports. Additional Information: Should pro sports leagues enjoy nonprofit status? Premise (Reason): Government is already struggling to pay for basic needs. And since  Warrant 0: the government isn’t required to pay for all the country’s needs  Warrant 1: the government is required to pay for the country’s needs Claim: Sport leagues should not enjoy nonprofit. Figure 1: Instance of the argument reasoning compre- hension task. The correct warrant has to be classified. Now, when we comprehend an argument, we re- construct its warrant driven by the cognitive princi- ple of relevance (Wilson and Sperber, 2004). What is easy for humans in many cases, however, turns out to be hard for machines, because reasoning usually depends on context and common sense. In (Habernal et al., 2018), we have thus introduced …

Content truncated to 500 words