IUCM at SemEval-2018 Task 11: Similar-Topic Texts as a Comprehension Knowledge Source

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 1068–1072 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics IUCM at SemEval-2018 Task 11: Similar-Topic Texts as a Comprehension Knowledge Source Sofia Reznikova Innopolis University Innopolis, Russia s.reznikova@innopolis.ru Leon Derczynski IT University of Copenhagen Denmark leod@itu.dk Abstract This paper describes the IUCM entry at SemEval-2018 Task 11, on machine com- prehension using commonsense knowledge. First, clustering and topic modeling are used to divide given texts into topics. Then, during the answering phase, other texts of the same topic are retrieved and used as commonsense knowl- edge. Finally, the answer is selected. While clustering itself shows good results, finding an answer proves to be more challenging. This paper reports the results of system evaluation and suggests potential improvements. 1 Introduction The goal of SemEval-2018 Task 11 is to find a way to incorporate commonsense knowledge into a question-answering task (Ostermann et al., 2018b). In this case, questions are either directly or indirectly related to given English texts; some questions may be answered using the text while others require background (commonsense) knowl- edge. The challenge is to use this knowledge in such a way as to enhance the quality of chosen an- swers. There are many approaches to question answer- ing including using structured knowledge (Yao and Durme, 2014), knowledge databases (Yih et al., 2015), deep learning methods (Minaee and Liu, 2017) and hybrid methods (Xu et al., 2016; Das et al., 2017). The present task accepts any method or any source of background knowledge. The training data consists of 1469 texts covering more than 100 topics. The number of questions per text varies from 1 to 14 and there are two an- swer options. The development data has 219 texts and the test data has 430. (Ostermann et al., 2018a) The main idea behind the method proposed in this paper is to use the given texts as potential sources of knowledge. Texts from training and development data can be divided into topics us- ing existing clustering algorithms (e.g. k-Means, Hierarchical, Grid-based or Density-based). The hypothesis is that texts which come from the same topic as the current question’s text may contain the correct answer. A matching function with a scor- ing scheme is used to identify the correct combi- nation of words for the answer. Another potential source of knowledge is scripts (Wanzare et al., 2016) that have been used to search for an answer. The DeScript dataset in- cludes descriptions of everyday activities such as baking, getting a haircut, going grocery shopping, and others, corresponding to topics present in the given data. Section 2 describes the methods as well as specifics of implementation. Section 3 provide in- terpretation and analysis of the results. Section 4 concludes the present paper. 2 Methodology All texts, questions and answers were tokenized, punctuation and extra symbols were removed. WordNet (Fellbaum, 1998) was used for lemma- tization using its morphy() function. Transform- ing all words into their initial form resulted in …

Content truncated to 500 words