Yuanfudao at SemEval-2018 Task 11: Three-way Attention and Relational Knowledge for Commonsense Machine Comprehension

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 758–762 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics Yuanfudao at SemEval-2018 Task 11: Three-way Attention and Relational Knowledge for Commonsense Machine Comprehension Liang Wang Meng Sun Wei Zhao Kewei Shen Jingming Liu Yuanfudao Research Beijing, China {wangliang01,sunmeng,zhaowei01,shenkw,liujm}@fenbi.com Abstract This paper describes our system for SemEval- 2018 Task 11: Machine Comprehension using Commonsense Knowledge (Oster- mann et al., 2018b). We use Three- way Attentive Networks (TriAN) to model interactions between the passage, question and answers. To incorporate commonsense knowledge, we augment the input with re- lation embedding from the graph of gen- eral knowledge ConceptNet (Speer et al., 2017). As a result, our system achieves state-of-the-art performance with 83.95% ac- curacy on the official test data. Code is pub- licly available at https://github.com/ intfloat/commonsense-rc. 1 Introduction It is well known that humans have a vast amount of commonsense knowledge acquired from everyday life. For machine reading comprehension, natu- ral language inference and many other NLP tasks, commonsense reasoning is one of the major obsta- cles to make machines as intelligent as humans. A large portion of previous work focus on com- monsense knowledge acquisition with unsuper- vised learning (Chambers and Jurafsky, 2008; Tandon et al., 2017) or crowdsourcing approach (Singh et al., 2002; Wanzare et al., 2016). Con- ceptNet (Speer et al., 2017), WebChild (Tan- don et al., 2017) and DeScript (Wanzare et al., 2016) etc are all publicly available knowledge re- sources. However, resources based on unsuper- vised learning tend to be noisy, while crowdsourc- ing approach has scalability issues. There is also some research on incorporating knowledge into NLP tasks such as reading comprehension (Lin et al., 2017; Yang and Mitchell, 2017) neural ma- chine translation (Zhang et al., 2017a) and text classification (Zhang et al., 2017b) etc. Though experiments show performance gains over base- lines, these gains are often quite marginal over the state-of-the-art system without external knowl- edge. In this paper, we present Three-way Attentive Networks(TriAN) for multiple-choice common- sense reading comprehension. The given task re- quires modeling interactions between the passage, question and answers. Different questions need to focus on different parts of the passage, attention mechanism is a natural choice and turns out to be effective for reading comprehension. Due to the relatively small size of training data, TriAN use word-level attention and consists of only one layer of LSTM (Hochreiter and Schmidhuber, 1997). Deeper models result in serious overfitting and poor generalization empirically. To explicitly model commonsense knowledge, relation embeddings based on ConceptNet (Speer et al., 2017) are used as additional features. ConceptNet is a large-scale graph of general knowledge from both crowdsourced resources and expert-created resources. It consists of over 21 million edges and 8 million nodes. ConceptNet shows state-of-the-art performance on tasks like word analogy and word relatedness. Besides, we also find that pretraining our net- work on other datasets helps to improve the overall performance. There are some existing multiple- choice English reading comprehension datasets contributed …

Content truncated to 500 words