Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3931 - 3943 December 7-11, 2022 ©2022 Association for Computational Linguistics Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models Shujian Zhang Chengyue Gong The University of Texas at AustinXingchao Liu {szhang19, cygong, xcliu}@utexas.edu Abstract Retriever-reader models achieve competitive performance across many different NLP tasks such as open question answering and dialogue conversations. In this work, we notice these models easily overfit the top-rank retrieval pas- sages and standard training fails to reason over the entire retrieval passages. We introduce a learnable passage mask mechanism which de- sensitizes the impact from the top-rank retrieval passages and prevents the model from overfit- ting. Controlling the gradient variance with fewer mask candidates and selecting the mask candidates with one-shot bi-level optimization, our learnable regularization strategy enforces the answer generation to focus on the entire retrieval passages. Experiments on different tasks across open question answering, dialogue conversation, and fact verification show that our method consistently outperforms its base- lines. Extensive experiments and ablation stud- ies demonstrate that our method can be general, effective, and beneficial for many NLP tasks. 1 Introduction Retriever-reader based approaches are popularly considered in the knowledge-intensive tasks (e.g., open Question Answering (QA), fact verification). It is designed to retrieve a set of support docu- ments and extract the answer from these documents. Mostly adopted retrieve and read models (e.g., Izac- ard and Grave, 2020) are trained to generate the annotated gold answers using the reader model, based on passages obtained by the retrievers (e.g., Robertson and Zaragoza, 2009; Karpukhin et al., 2020). This training process of reader disregards the evidentiality of all retrieval passages and can easily overfit the top ranked passages (Xu et al., 2021; Lee et al., 2021). Even if the top-rank pas- sages in the test setting do not have the correct answers, these models still tend to find the answer in the top-rank passages and yield worse perfor- mance (Xu et al., 2021). It happens to the reader model due to the overfitting and the memorization of outdated information (Longpre et al., 2021). To what extent does the reader model quality depend on the retrieval passages? We analyze the ranking impact of the retrieval passages from mask- ing (e.g. mask out the top three passages), permut- ing, and removing. The overfitting, as well as the performance degradation, is observed. To desen- sitize the impact from the top-rank passages, we consider masking passages during training which serves as a desensitizer and can improve the reader model ability to reason over all retrieval passages. However, the standard masking and dropout strategies are not designed for our focused tasks and also bring an increased gradient variance dur- ing training due to their randomness. In the mean- time, each neuron plays the same role and has the same mask. However, in the reader model, in- tuitively, top-rank passages often have a higher chance to overfit during the training. To this end, we introduce our passage mask (PM), …

Content truncated to 500 words