Hurdles to Progress in Long-form Question Answering

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4940–4957 June 6–11, 2021. ©2021 Association for Computational Linguistics 4940 Hurdles to Progress in Long-form Question Answering Kalpesh Krishna♠ ∗ Aurko Roy♦ ♠University of Massachusetts Amherst, ♦Google Research {kalpesh,miyyer}@cs.umass.edu aurkor@google.com Mohit Iyyer♠ Abstract The task of long-form question answering (LFQA) involves retrieving documents rele- vant to a given question and using them to generate a paragraph-length answer. While many models have recently been proposed for LFQA, we show in this paper that the task formulation raises fundamental chal- lenges regarding evaluation and dataset cre- ation that currently preclude meaningful mod- eling progress. To demonstrate these chal- lenges, we first design a new system that relies on sparse attention and contrastive re- triever learning to achieve state-of-the-art per- formance on the ELI5 LFQA dataset. While our system tops the public leaderboard, a de- tailed analysis reveals several troubling trends: (1) our system’s generated answers are not ac- tually grounded in the documents that it re- trieves; (2) ELI5 contains significant train / val- idation overlap, as at least 81% of ELI5 vali- dation questions occur in paraphrased form in the training set; (3) ROUGE-L is not an infor- mative metric of generated answer quality and can be easily gamed; and (4) human evalua- tions used for other text generation tasks are unreliable for LFQA. We offer suggestions to mitigate each of these issues, which we hope will lead to more rigorous LFQA research and meaningful progress in the future.1 1 Introduction Long-form question answering (LFQA) integrates the retrieval component of open-domain QA, which involves searching a large external knowl- edge source for documents relevant to a given ques- tion, with a text generation component to produce paragraph-length answers. Significant progress has been made on open-domain QA datasets such as Natural Questions (Kwiatkowski et al., 2019), * Work done during an internship at Google Research. 1Resources accompanying our paper can be found in https://github.com/martiansideofthemoon/ hurdles-longform-qa whose questions are answerable with short phrases and entities, by leveraging dense retrieval tech- niques like ORQA (Lee et al., 2019), REALM (Guu et al., 2020), and DPR (Karpukhin et al., 2020; Lewis et al., 2020c; Izacard and Grave, 2020). Methods inspired by these results have recently been combined with pretrained language mod- els (Lewis et al., 2020b; Petroni et al., 2020) and applied to the Reddit-derived “Explain Like I’m Five” (ELI5) dataset (Fan et al., 2019), which is the only publicly-available large-scale LFQA dataset. The recently proposed KILT benchmark (Petroni et al., 2020), which compares retrieval-augmented models across a variety of knowledge-intensive tasks including ELI5, automatically evaluates LFQA models by the quality of both generated an- swers (ROUGE-L against reference answers) and retrieved documents (R-precision against human- annotated relevant documents). In this paper, we build a state-of-the-art system2 for ELI5 by using a sparse Transformer variant (Roy et al., 2020) to condition over Wikipedia paragraphs returned by a REALM-style retriever (Guu et al., 2020). However, despite …

Content truncated to 500 words