HHU at SemEval-2018 Task 12: Analyzing an Ensemble-based Deep Learning Approach for the Argument Mining Task of Choosing the Correct Warrant

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 1114–1119 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics HHU at SemEval-2018 Task 12: Analyzing an Ensemble-based Deep Learning Approach for the Argument Mining Task of Choosing the Correct Warrant Matthias Liebeck, Andreas Funke, Stefan Conrad Institute of Computer Science Heinrich Heine University D¨usseldorf D-40225 D¨usseldorf, Germany {liebeck,conrad}@cs.uni-duesseldorf.de andreas.funke@hhu.de Abstract This paper describes our participation in the SemEval-2018 Task 12 Argument Reasoning Comprehension Task which calls to develop systems that, given a reason and a claim, pre- dict the correct warrant from two opposing op- tions. We decided to use a deep learning archi- tecture and combined 623 models with differ- ent hyperparameters into an ensemble. Our ex- tensive analysis of our architecture and ensem- ble reveals that the decision to use an ensemble was suboptimal. Additionally, we benchmark a support vector machine as a baseline. Fur- thermore, we experimented with an alternative data split and achieved more stable results. 1 Introduction Argument mining is a trending research do- main that focuses on the extraction of arguments and their relations from text. It has been ap- plied to multiple languages and multiple applica- tion domains, including the legal domain (Palau and Moens, 2009), persuasive essays (Stab and Gurevych, 2014), online participation (Liebeck et al., 2016), and web discourse (Habernal and Gurevych, 2017). The three most common sub- tasks are: argument identification, argument clas- sification, and argument linking. These focus on the identification of argumentative text content, the extraction of argument components according to a specific argument model, and the extraction of relations between arguments components, respec- tively. Currently, different argument models compris- ing different argument components are being used throughout the community, such as the claim- premise family or Toulmin’s model (Toulmin, 1958). In the scope of this paper, claims, premises, and warrants are the most important argument components. Claims are often defined as con- troversial statements that are either true or false. Premises are reasons that support or attack claims. In Toulmin’s model, they are connected with war- rants that state why the premise supports the claim. Habernal et al. (2018b) introduced a new argu- ment mining task called argument reasoning com- prehension with the following definition: Given a reason and a claim, identify the correct warrant from two opposing options. This paper describes our participation in the SemEval-2018 Task 12 The Argument Reasoning Comprehension Task (Habernal et al., 2018a) that uses the dataset from Habernal et al. (2018b) as a shared task. Besides a description of our machine learning systems, we evaluate additional machine learning models (we were only allowed to submit a single set of predictions for the official ranking) and we further analyze the test set. The dataset for the challenge consists of anno- tated news comments from the New York Times user debate section. With Amazon Mechanical Turk as a crowdsourcing platform, 5000 randomly selected user comments were annotated in a multi- step annotation process that included three free text annotation …

Content truncated to 500 words