Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3170 - 3179 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, Michael Zeng Microsoft Azure Cognitive Services Research {shuowa, yicxu, yuwfan, yaliu10, siqisun, ruox, chezhu, nzeng}@microsoft.com Abstract Retrieval-based methods have been shown to be effective in NLP tasks via introducing external knowledge. However, the indexing and retriev- ing of large-scale corpora bring considerable computational cost. Surprisingly, we found that REtrieving from the traINing datA (REINA) only can lead to significant gains on multiple NLG and NLU tasks. We retrieve the labeled training instances most similar to the input text and then concatenate them with the input to feed into the model to generate the output. Ex- perimental results show that this simple method can achieve significantly better performance on a variety of NLU and NLG tasks, including summarization, machine translation, language modeling, and question answering tasks. For in- stance, our proposed method achieved state-of- the-art results on XSum, BigPatent, and Com- monsenseQA. Our code is released.1 1 Introduction In natural language processing, retrieval-based methods work by fetching textual information re- lated to the input from large corpora. The model then takes both the input and retrieved results as input to generate results. This can often improve the performance as the model is exposed to related knowledge not present in the input. As a result, retrieval-based methods have been successfully ap- plied in many tasks such as open-domain question answering (Chen et al., 2017), language model- ing (Guu et al., 2018; Khandelwal et al., 2020) and machine translation (Khandelwal et al., 2021). However, these methods require building an index of large-scale corpus, and the retrieval leads to a significant computational burden. For example, the kNN-MT model for machine translation has a gen- eration speed two orders of magnitude slower than traditional MT models (Khandelwal et al., 2021). On the other hand, in the supervised learning setting, the text most similar in distribution to the 1https://github.com/microsoft/REINA Figure 1: REINA pipeline of model training/inference with retrieval from training data. Filter only happens at training, as the same training sample will be retrieved from the index. For each instance, we concatenate the input with the retrieved content, i.e., data and/or labels, for model training and inference. data in inference is the training data. Thus, we explore whether retrieving from the training data, which is usually much smaller than a large-scale corpus, can help improve the performance. Specif- ically, we first index a task’s labeled training data as input-label pairs. Then, during both training and testing, we retrieve the input-label pairs most sim- ilar to the current input2. Finally, we concatenate the retrieved training pairs with the input and feed it into the model. An overview of our method is shown in Figure 1. We note …

Content truncated to 500 words