Jiangnan at SemEval-2018 Task 11: Deep Neural Network with Attention Method for Machine Comprehension Task

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 1063–1067 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics Jiangnan at SemEval-2018 Task 11: Deep Neural Network with Attention Method for Machine Comprehension Task Jiangnan Xia Alibaba Group Hangzhou, China jiangnan xjn@alibaba-inc.com Abstract This paper describes our submission for the International Workshop on Semantic Evalua- tion (SemEval-2018) shared task 11– Machine Comprehension using Commonsense Knowl- edge (Ostermann et al., 2018b). We use a deep neural network model to choose the cor- rect answer from the candidate answers pair when the document and question are given. The interactions between document, question and answers are modeled by attention mech- anism and a variety of manual features are used to improve model performance. We also use CoVe (McCann et al., 2017) as an exter- nal source of knowledge which is not men- tioned in the document. As a result, our sys- tem achieves 80.91% accuracy on the test data, which is on the third place of the leaderboard. 1 Introduction In recent years, machine reading comprehension (MRC) which attempts to enable machines to an- swer questions when given a set of documents, has attracted great attentions. Several MRC datasets have been released such as the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016) and the Microsoft MAchine Reading COm- prehension Dataset (MS-MARCO) (Nguyen et al., 2016). These datasets provide large scale of man- ually created data, greatly inspired the research in this field. And a series of neural network model, such as BiDAF (Seo et al., 2016), R-Net (Wang et al., 2017), have achieved promising results on these evaluation tasks. However, machine read- ing comprehension is still a difficult task because without knowledge, machines cannot really under- stand the question and make a correct answer. As an effort to discover how machine read- ing comprehension systems would be benefited from commonsense knowledge, (Ostermann et al., 2018b) developed the Machine Comprehension using Commonsense Knowledge task. In this task, commonsense knowledge is given as the form of script knowledge. Script knowledge is defined as the knowledge about everyday activities which is mentioned in narrative documents. For each doc- ument, a series questions are asked and each ques- tion is associated with a pair of candidate answers. Machines have to choose which is the correct an- swer. To let machines make correct decisions, ex- plicit information which can be found in the doc- ument and external commonsense knowledge are both required. Table 1 shows an example of the dataset in this task. In this paper, we make a description about our submission system for the task. The system is based on a deep neural network model. The input of the model is a (document, question, answer) triple and the output is the probability that the an- swer is the correct one for the given document and question. We also combine the neural network model with a variety of manual features, including word exact match features and token features such as part-of-speech …

Content truncated to 500 words