ASQA: Factoid Questions Meet Long-Form Answers

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8273 - 8288 December 7-11, 2022 ©2022 Association for Computational Linguistics ASQA: Factoid Questions Meet Long-Form Answers Ivan Stelmakh1∗ Yi Luan3 Bhuwan Dhingra2,3 Ming-Wei Chang3 1Yakov & Partners 2Duke University 3Google Research stelmakh95@icloud.com {luanyi,bdhingra,mingweichang}@google.com Abstract An abundance of datasets and availability of re- liable evaluation metrics have resulted in strong progress in factoid question answering (QA). This progress, however, does not easily transfer to the task of long-form QA, where the goal is to answer questions that require in-depth explanations. The hurdles include (i) a lack of high-quality data, and (ii) the absence of a well-defined notion of the answer’s quality. In this work, we address these problems by (i) releasing a novel dataset and a task that we call ASQA (Answer Summaries for Questions which are Ambiguous); and (ii) proposing a reliable metric for measuring performance on ASQA. Our task focuses on factoid questions that are ambiguous, that is, have different cor- rect answers depending on interpretation. An- swers to ambiguous questions should synthe- size factual information from multiple sources into a long-form summary that resolves the am- biguity. In contrast to existing long-form QA tasks (such as ELI5), ASQA admits a clear notion of correctness: a user faced with a good summary should be able to answer different interpretations of the original ambiguous ques- tion. We use this notion of correctness to de- fine an automated metric of performance for ASQA. Our analysis demonstrates an agree- ment between this metric and human judg- ments, and reveals a considerable gap between human performance and strong baselines. 1 Introduction In the last few years, the factoid question answer- ing (QA) task—extracting short answers to factoid questions—has witnessed significant progress (Lee et al., 2019; Guu et al., 2020; Karpukhin et al., 2020; Lewis et al., 2020; Izacard and Grave, 2021). The progress was achieved in large part thanks to (i) the availability of high-quality datasets (Voorhees and Tice, 2000; Joshi et al., 2017; Yang et al., 2018; Abujabal et al., 2019; Kwiatkowski et al., 2019), ∗Work done during an internship at Google Research. and (ii) a well-defined notion of correctness. A key challenge for ongoing research now lies in long- form question answering where the goal is to gen- erate detailed explanations in response to questions that require elaborate and in-depth answers. There is much less data available for the task of long-form QA. One of the primary data sources is the ELI5 dataset (Fan et al., 2019) that pairs open-ended questions with paragraph-long answers written by users of the “Explain Like I’m Five” Reddit forum. However, questions in ELI5 are very general (e.g., “How can different animals perceive different colors?”) and can be an- swered in myriad different ways, making it hard to define objective criteria for a good answer. As a re- sult, Krishna et al. (2021) identify several hurdles in using this data towards meaningful modeling progress, including a lack of reliable evaluation metrics. In …

Content truncated to 500 words