NLITrans at SemEval-2018 Task 12: Transfer of Semantic Knowledge for Argument Comprehension

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 1099–1103 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics NLITrans at SemEval-2018 Task 12: Transfer of Semantic Knowledge for Argument Comprehension Timothy Niven and Hung-Yu Kao National Cheng Kung University Tainan, Taiwan tim.niven.public@gmail.com, hykao@mail.ncku.edu.tw Abstract The Argument Reasoning Comprehension Task requires significant language understand- ing and complex reasoning over world knowl- edge. We focus on transfer of a sentence encoder to bootstrap more complicated mod- els given the small size of the dataset. Our best model uses a pre-trained BiLSTM to en- code input sentences, learns task-specific fea- tures for the argument and warrants, then per- forms independent argument-warrant match- ing. This model achieves mean test set accu- racy of 64.43%. Encoder transfer yields a sig- nificant gain to our best model over random initialization. Independent warrant matching effectively doubles the size of the dataset and provides additional regularization. We demon- strate that regularization comes from ignoring statistical correlations between warrant fea- tures and position. We also report an exper- iment with our best model that only matches warrants to reasons, ignoring claims. Rela- tively low performance degradation suggests that our model is not necessarily learning the intended task. 1 Introduction The Argument Reasoning Comprehension Task (ARCT) (Habernal et al., 2018) addresses a sig- nificant open problem in argumentation mining: connecting reasons and claims via inferential li- censes, called warrants (Toulmin, 1958). War- rants are a form of shared world knowledge and are mostly implicit in argumentation. This makes it difficult for machine learning algorithms to dis- cover arguments, as they must acquire and use this knowledge to identify argument components and their relations. ARCT isolates the reasoning step by not requiring warrants to be discovered. A cor- rect warrant W and an incorrect alternative A are given, and the correct one must be predicted given the corresponding claim C and reason R. Figure 1: Benefit of transfer to our best model, COMP. Distributions come from 200 runs with different ran- dom seeds. Mean accuracy for transfer (64.43%) is higher than random (61.81%) and is significant with p = 9.68 × 10−41. However, this does not eliminate the need for other forms of world knowledge. Consider the fol- lowing example from the test set: C Google is not a harmful monopoly R People can choose not to use Google W Other search engines do not re-direct to Google A All other search engines re-direct to Google It is required to know how consumer choice and web re-directs relate to the concept of monopoly in this context, and that Google is a search engine. We do not attempt to address these other forms of world knowledge. Given the small size of the dataset we focus on transfer of semantic knowl- edge in the form of a sentence encoder to boot- strap inference over learned features. Following Conneau et al. (2017), we pre-train a BiLSTM encoder with max pooling on natural language in- ference (NLI) data (Williams et …

Content truncated to 500 words