Relation Module for Non-Answerable Predictions on Reading Comprehension

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 747–756 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 747 Relation Module for Non-answerable Prediction on Reading Comprehension Kevin Huang, Yun Tang, Jing Huang, Xiaodong He, and Bowen Zhou JD AI Research, Mountain View, CA {kevin.huang3, yun.tang, jing.huang, xiadong.he, bowen.zhou}@jd.com Abstract Machine reading comprehension (MRC) has attracted significant amounts of research atten- tion recently, due to an increase of challenging reading comprehension datasets. In this pa- per, we aim to improve a MRC model’s abil- ity to determine whether a question has an an- swer in a given context (e.g. the recently pro- posed SQuAD 2.0 task). Our solution is a re- lation module that is adaptable to any MRC model. The relation module consists of both semantic extraction and relational information. We first extract high level semantics as objects from both question and context with multi- head self-attentive pooling. These semantic objects are then passed to a relation network, which generates relationship scores for each object pair in a sentence. These scores are used to determine whether a question is non- answerable. We test the relation module on the SQuAD 2.0 dataset using both the BiDAF and BERT models as baseline readers. We ob- tain 1.8% gain of F1 accuracy on top of the BiDAF reader, and 1.0% on top of the BERT base model. These results show the effective- ness of our relation module on MRC. 1 Introduction Ever since the release of many challenging large scale datasets for machine reading comprehension (MRC) (Rajpurkar et al., 2016; Joshi et al., 2017; Trischler et al., 2016; Yang et al., 2018; Reddy et al., 2018; Jia and Liang, 2017), there have been correspondingly many models for these datasets (Yu et al., 2018; Seo et al., 2017; Liu et al., 2018b; Hu et al., 2017; Xiong et al., 2017; Wang et al., 2018; Liu et al., 2018c; Tay et al., 2018). Know- ing what you don’t know (Rajpurkar et al., 2018) is important in real applications of reading compre- hension. Unanswerable questions are common- place in the real world, and SQuAD 2.0 was re- leased specifically to target this problem (see Fig- ure 1 for an example of non-answerable ques- tions). Example 1 Context: Each year, the southern California area has about 10,000 earthquakes. Nearly all of them are so small that they are not felt. Only several hundred are greater than magnitude 3.0, and only about 1520 are greater than magnitude 4.0. The magnitude 6.7 1994 Northridge earthquake was particularly destructive, causing a sub- stantial number of deaths, injuries, and structural col- lapses. It caused the most property damage of any earth- quake in U.S. history, estimated at over $20 billion. Question: What earthquake caused $20 million in dam- age? Answer: None. Figure 1: An example of non-answerable question in SQuAD 2.0. Highlighted words are the output from the BERT base model. The true answer is “None”. One problem that most of the early MRC read- …

Content truncated to 500 words