Developing Dataset of Japanese Slot Filling Quizzes Designed for Evaluation of Machine Reading Comprehension

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6895–6901 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6895 Developing Dataset of Japanese Slot Filling Quizzes Designed for Evaluation of Machine Reading Comprehension Takuto Watarai, Masatoshi Tsuchiya Toyohashi University of Technology 1–1 Hibarigaoka, Tempaku-cho, Toyohashi, Aichi, Japan {watarai, tsuchiya}@is.cs.tut.ac.jp Abstract This paper describes our developing dataset of Japanese slot filling quizzes designed for evaluation of machine reading comprehension. The dataset consists of quizzes automatically generated from Aozora Bunko, and each quiz is defined as a 4-tuple: a context passage, a query holding a slot, an answer character and a set of possible answer characters. The query is generated from the original sentence, which appears immediately after the context passage on the target book, by replacing the answer character into the slot. The set of possible answer characters consists of the answer character and the other characters who appear in the context passage. Because the context passage and the query shares the same context, a machine which precisely understand the context may select the correct answer from the set of possible answer characters. The unique point of our approach is that we focus on characters of target books as slots to generate queries from original sentences, because they play important roles in narrative texts and precise understanding their relationship is necessary for reading comprehension. To extract characters from target books, manually created dictionaries of characters are employed because some characters appear as common nouns not as named entities. Keywords: Machine Reading Comprehension, Slot Filling Quiz, Novel Characters 1. Introduction Machine reading comprehension is one of crucial goals of the field of natural language processing. To achieve this goal directly, it is necessary to define what reading com- prehension is, however it is quite difficult. Therefore, in- stead of defining reading comprehension directly, evalua- tion based on benchmark tasks, which can be achieved if and only if target sentences are precisely understood, are widely employed. There are three typical benchmark tasks widely em- ployed for evaluation of machine reading comprehen- sion. The first one is the slot filling task (Hill et al., 2016; Hermann et al., 2015) which is defined as follows. Suppose n + 1 sentences si+n+1 i which appear sequentially in a tar- get text. When a context passage C which consists of n sentences si+n i and a query q generated from the sentence si+n+1, which appears immediately after the passage C, by replacing a word into a slot, and a set of possible an- swer words A are given, a machine must select the cor- rect answer word a, which appears in the original sentence si+n+1, from the set A. The second one is the 4 multiple choice question task (Richardson et al., 2013) which is de- fined as follows: when a passage C, a query q about a fact described in the passage C and a set of possible answer sentences A are given, a machine must select …

Content truncated to 500 words