Answerable or Not: Devising a Dataset for Extending Machine Reading Comprehension

Proceedings of the 27th International Conference on Computational Linguistics, pages 973–983 Santa Fe, New Mexico, USA, August 20-26, 2018. 973 Answerable or Not: Devising a Dataset for Extending Machine Reading Comprehension Mao Nakanishi Tetsunori Kobayashi Yoshihiko Hayashi School of Science and Engineering, Waseda University Waseda-machi 27, Shinjuku, Tokyo 1690042, Japan nakanishi@pcl.cs.waseda.ac.jp koba@waseda.jp yshk.hayashi@aoni.waseda.jp Abstract Machine reading comprehension (MRC) has recently attracted attention in the fields of natural language processing and machine learning. One of the problematic presumptions with current MRC technologies is that each question is assumed to be answerable by looking at a given text passage. However, to realize human-like language comprehension ability, a machine should also be able to distinguish not-answerable questions (NAQs) from answerable questions. To develop this functionality, a dataset incorporating hard-to-detect NAQs is vital; however, its manual con- struction would be expensive. This paper proposes a dataset creation method that alters an exist- ing MRC dataset, the Stanford Question Answering Dataset, and describes the resulting dataset. The value of this dataset is likely to increase if each NAQ in the dataset is properly classified with the difficulty of identifying it as an NAQ. This difficulty level would allow researchers to evalu- ate a machine’s NAQ detection performance more precisely. Therefore, we propose a method for automatically assigning difficulty level labels, which basically measures the similarity between a question and the target text passage. Our NAQ detection experiments demonstrate that the result- ing dataset, having difficulty level annotations, is valid and potentially useful in the development of advanced MRC models. 1 Introduction Language understanding is one of the ultimate goals of artificial intelligence. Because the machine understanding of human language is hard to define, one often presumes that if a machine can answer a set of questions under given circumstances, then it understands language. Machine reading comprehension (MRC) developed along this direction and has recently attracted a great deal of attention in natural language processing (NLP) and machine learning. The tasks of MRC are defined in various ways (Richardson et al., 2013; Mostafazadeh et al., 2017; Weston et al., 2015; Chen et al., 2016; Rajpurkar et al., 2016). Some tasks (Chen et al., 2016; Hill et al., 2016) provide a set of fill-in-the-blank style questions. The most prominent tasks (Rajpurkar et al., 2016), however, allow a machine to locate an answer span in given text passages. It is a highly efficient scheme, because the task can be formulated by the recognition of the start and end positions of the passage. This model facilitates the development of machine learning approaches and the automatic evaluation of performances. However, as criticized by (Jia and Liang, 2017), a machine can succeed in a task of this kind by remembering and recalling linguistic patterns that are prominent in a target dataset. This means that machine comprehension of this type would be an insufficient measure for assessing the degree of machine understanding of language. To tackle this issue, Jia and Liang (2017) developed a dataset, in which passages were altered to include …

Content truncated to 500 words