Generate-and-Retrieve: Use Your Predictions to Improve Retrieval for Semantic Parsing

Proceedings of the 29th International Conference on Computational Linguistics, pages 4946–4951 October 12–17, 2022. 4946 Generate-and-Retrieve: use your predictions to improve retrieval for semantic parsing Yury Zemlyanskiy∗ † Michiel de Jong∗ † Joshua Ainslie‡ Panupong Pasupat‡ Peter Shaw‡ Linlu Qiu‡ Sumit Sanghai‡ Fei Sha‡ † University of Southern California ‡ Google Research {yury.zemlyanskiy,msdejong}@usc.edu {jainslie,ppasupat,petershaw,linluqiu,sumitsanghai,fsha}@google.com Abstract A common recent approach to semantic pars- ing augments sequence-to-sequence models by retrieving and appending a set of training samples, called exemplars. The effectiveness of this recipe is limited by the ability to re- trieve informative exemplars that help produce the correct parse, which is especially challeng- ing in low-resource settings. Existing retrieval is commonly based on similarity of query and exemplar inputs. We propose GandR, a re- trieval procedure that retrieves exemplars for which outputs are also similar. GandR first generates a preliminary prediction with input- based retrieval. Then, it retrieves exemplars with outputs similar to the preliminary predic- tion which are used to generate a final predic- tion. GandR sets the state of the art on multiple low-resource semantic parsing tasks. 1 Introduction A common and successful approach to structured prediction problems (Li et al., 2021; Chen et al., 2020) is to treat the gold structure as a sequence and fine-tune a sequence-to-sequence model such as T5 (Raffel et al., 2020) or BART (Lewis et al., 2020). However, the performance of fine-tuned models suffers in low resource scenarios where available training data is limited relative to the complexity of the task (Chen et al., 2020). Existing work (Pasupat et al., 2021; Gupta et al., 2021; Wang et al., 2022) has found that retrieving related training samples, denoted exemplars, and appending the retrieved input-output pairs to the sample input before processing the sample can im- prove performance in low resource settings. In prin- ciple, all information from exemplars is available to the model during training and could be stored in model parameters. However, in practice the model ∗Work primarily done at Google Research. may not successfully retain all information, and re- minding the model of salient input-output patterns at test time appears to help. That raises the question: what exemplars are most informative for the model? Existing work fo- cuses on retrieving exemplars for which the input is partially similar to the test input, effectively answer- ing “What is the output for similar inputs?”. In this work we explore whether there is complementary information in exemplars that answer the inverse question, “What is the input for similar outputs?”. We propose Generate-and-Retrieve (GandR), a method to retrieve exemplars with similar output as well as input. As the true output of a sample is in general unknown, GandR proceeds in two steps. First, a preliminary prediction is generated using retrievals with similar input only. Then, a new set of exemplars is retrieved based on a relevance measure that balances the similarity of the inputs and the similarity of the preliminary prediction and the exemplar output. Figure 1 provides an overview of the method. We evaluate GandR …

Content truncated to 500 words