Query Refinement Prompts for Closed-Book Long-Form QA

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7997–8012 July 9-14, 2023 ©2023 Association for Computational Linguistics Query Refinement Prompts for Closed-Book Long-Form QA Reinald Kim Amplayo Kellie Webster Michael Collins Dipanjan Das Shashi Narayan Google DeepMind {reinald, websterk, mjcollins, dipanjand, shashinarayan}@google.com Abstract Large language models (LLMs) have been shown to perform well in answering questions and in producing long-form texts, both in few- shot closed-book settings. While the former can be validated using well-known evaluation metrics, the latter is difficult to evaluate. We resolve the difficulties to evaluate long-form output by doing both tasks at once – to do ques- tion answering that requires long-form answers. Such questions tend to be multifaceted, i.e., they may have ambiguities and/or require infor- mation from multiple sources. To this end, we define query refinement prompts that encourage LLMs to explicitly express the multifaceted- ness in questions and generate long-form an- swers covering multiple facets of the question. Our experiments on two long-form question answering datasets, ASQA and AQuAMuSe, show that using our prompts allows us to out- perform fully finetuned models in the closed book setting, as well as achieve results compa- rable to retrieve-then-generate open-book mod- els. 1 Introduction Interest in large language models (LLMs) has exploded in recent years due to their wide range of abilities im- proving the state-of-the-art with just a few or no exam- ples (Brown et al., 2020; Wei et al., 2022a; Chowdhery et al., 2022; Wei et al., 2022b). One task that bene- fited greatly is closed-book question answering (Roberts et al., 2020) in a few-shot setting, i.e., to produce cor- rect answers to questions without access to passages to read and find the answer. Most of the impressive re- sults, however, are limited to generating short answers, and while previous work has utilized LLMs to gener- ate long-form texts (Elkins and Chun, 2020; Wei et al., 2022b), most of these outputs are difficult to evaluate since they are subjective in nature (i.e., they may have multiple correct and distinctive answers). In this paper, we attempt to evaluate the long-form generation ability of LLMs through long-form question answering (Fan et al., 2019) in the closed-book setting. Unlike question answering with short answers, ques- tions with long-form answers are naturally multifacted, covering many aspects that are required to fully an- swer the questions (see Table 1 for examples). This introduces several challenges that LLMs would need to resolve: 1. Ambiguity: Many questions in a real-world set- ting are ambiguous in some way; Min et al. (2020) estimated that more than half of Natural Questions (Kwiatkowski et al., 2019) have multiple plausible interpretations. In the first example of Table 1, the question is ambiguous owing to there being multi- ple types of cars. A system producing a long-form answer to this question should present the multiple answers in a coherent natural language text. 2. Information Consolidation: Some questions re- quire consolidating information from multiple sources to be able …

Content truncated to 500 words