Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1492–1504 August 1–6, 2021. ©2021 Association for Computational Linguistics 1492 Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval Akari Asai University of Washington akari@cs.washington.edu Eunsol Choi∗ The University of Texas at Austin eunsol@cs.utexas.edu Abstract Recent pretrained language models “solved” many reading comprehension benchmarks, where questions are written with the access to the evidence document. However, datasets containing information-seeking queries where evidence documents are provided after the queries are written independently remain challenging. We analyze why answering information-seeking queries is more challeng- ing and where their prevalent unanswerabili- ties arise, on Natural Questions and TyDi QA. Our controlled experiments suggest two head- rooms – paragraph selection and answerabil- ity prediction, i.e. whether the paired evidence document contains the answer to the query or not. When provided with a gold paragraph and knowing when to abstain from answering, ex- isting models easily outperform a human an- notator. However, predicting answerability it- self remains challenging. We manually an- notate 800 unanswerable examples across six languages on what makes them challenging to answer. With this new data, we conduct per- category answerability prediction, revealing is- sues in the current dataset collection as well as task formulation. Together, our study points to avenues for future research in information- seeking question answering, both for dataset creation and model development.1 1 Introduction Addressing the information needs of users by an- swering their questions can serve a variety of prac- tical applications. To answer such information- seeking queries – where users pose a question be- cause they do not know the answer – in an un- constrained setting is challenging for annotators as they have to exhaustively search over the web. 1Our code and annotated data is publicly avail- able at https://github.com/AkariAsai/ unanswerable_qa. To reduce annotator burden, the task has been sim- plified as reading comprehension: annotators are tasked with finding an answer in a single document. Recent pretrained language models surpassed es- timated human performance (Liu et al., 2019; De- vlin et al., 2019) in many reading comprehension datasets such as SQuAD (Rajpurkar et al., 2016) and CoQA (Reddy et al., 2019), where questions are posed with an answer in mind. However, those state-of-the-art models have difficulty answering information-seeking questions (Kwiatkowski et al., 2019; Choi et al., 2018). In this work, we investigate what makes information-seeking question answering (QA) more challenging, focusing on the Natural Ques- tions (NQ; Kwiatkowski et al., 2019) and TyDi QA (Clark et al., 2020) datasets. Our experimen- tal results from four different models over six lan- guages on NQ and TyDi QA show that most of their headroom can be explained by two subprob- lems: selecting a paragraph that is relevant to a question and deciding whether the paragraph con- tains an answer. The datasets are annotated at the document level, with dozens of paragraphs, and finding the correct paragraph is nontrivial. When provided with a gold paragraph …

Content truncated to 500 words