Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study

Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study Xiangyang Mou∗ Chenghao Yang∗ Mo Yu∗ Bingsheng Yao Xiaoxiao Guo Saloni Potdar Hui Su Rensselaer Polytechnic Institute & IBM, United States moux4@rpi.edu gflfof@gmail.com Abstract Recent advancements in open-domain ques- tion answering (ODQA), that is, finding an- swers from large open-domain corpus like Wikipedia, have led to human-level perfor- mance on many datasets. However, progress in QA over book stories (Book QA) lags de- spite its similar task formulation to ODQA. This work provides a comprehensive and quantitative analysis about the difficulty of Book QA: (1) We benchmark the research on the NarrativeQA dataset with extensive experiments with cutting-edge ODQA tech- niques. This quantifies the challenges Book QA poses, as well as advances the published state-of-the-art with a ∼7% absolute improve- ment on ROUGE-L. (2) We further analyze the detailed challenges in Book QA through human studies.1 Our findings indicate that the event-centric questions dominate this task, which exemplifies the inability of existing QA models to handle event-oriented scenarios. 1 Introduction Recent Question-Answering (QA) models have achieved or even surpassed human performance on many challenging tasks, including single- passage QA2 and open-domain QA (ODQA).3 Nevertheless, understanding rich context beyond text pattern matching remains unsolved, espe- ∗Equal contribution. XM built the whole system, imple- mented the data preprocessing pipeline, Hard EM ranker, and all the reader modules, and conducted all the QA exper- iments. CY implemented the unsupervised ICT ranker and the first working version of FiD, and was responsible for the final ranker module. MY is the corresponding author, who proposed and led this project, built the ranker code base (until the DS ranker), designed the question schema and conducted its related experiments and analysis in Part II. 1https://github.com/gorov/BookQA. 2The SQuAD leaderboard (Rajpurkar et al., 2018): rajpurkar.github.io/SQuAD-explorer. 3Wang et al. (2020); Iyer et al. (2020)’s results on Quasar-T (Dhingra et al., 2017) and SearchQA (Dunn et al., 2017). cially answering questions on narrative elements via reading books. One example is NarrativeQA (Koˇcisk`y et al., 2018) (Figure 1). Since its first release in 2017, there has been no significant im- provement over the primitive baselines. In this paper, we study this challenging Book QA task and shed light on the inherent difficulties. Despite its similarity to standard ODQA tasks,4 that is, both requiring finding evidence paragraphs for inferring answers, the Book QA has certain unique challenges (Koˇcisk`y et al., 2018): (1) The narrative writing style of book stories dif- fers from the formal texts in Wikipedia and news, which demands a deeper understanding capability. The flexible writing styles from different genres and authors make the challenge severe; (2) The passages that depict related book plots and char- acters share more semantic similarities than the Wikipedia articles, which increases confusion in finding the correct evidence to answer a question; (3) The free-form nature of the answers necessi- tates the summarization ability from the narrative plots; (4) The free-form answers make it hard to obtain fine-grained supervision at passage or span levels; and …

Content truncated to 500 words