FiE: Building a Global Probability Space by Leveraging Early Fusion in Encoder for Open-Domain Question Answering

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4246 - 4260 December 7-11, 2022 ©2022 Association for Computational Linguistics FiE: Building a Global Probability Space by Leveraging Early Fusion in Encoder for Open-Domain Question Answering Akhil Kedia and Mohd Abbas Zaidi and Haejun Lee Samsung Research, Seoul {akhil.kedia, abbas.zaidi, haejun82.lee}@samsung.com Abstract Generative models have recently started to outperform extractive models in Open Do- main Question Answering, largely by leverag- ing their decoder to attend over multiple en- coded passages and combining their informa- tion. However, generative models tend to be larger than extractive models due to the need for a decoder, run slower during inference due to auto-regressive decoder beam search, and their generated output often suffers from hal- lucinations. We propose to extend transformer encoders with the ability to fuse information from multiple passages, using global represen- tation to provide cross-sample attention over all tokens across samples. Furthermore, we propose an alternative answer span probability calculation to better aggregate answer scores in the global space of all samples. Using our pro- posed method, we outperform the current state- of-the-art method by 2.5 Exact Match score on the Natural Question dataset while using only 25% of parameters and 35% of the latency during inference, and 4.4 Exact Match on We- bQuestions dataset. When coupled with syn- thetic data augmentation, we outperform larger models on the TriviaQA dataset as well. The latency and parameter savings of our method make it particularly attractive for open-domain question answering, as these models are often compute-intensive. 1 Introduction Open-Domain Question-Answering is the task of answering an input question given a large external knowledge-base, such as the entire Wikipedia. This problem is typically approached by leveraging a retriever model to first retrieve a set of relevant documents/passages using some IR method, which are then passed on to a reader model (Lee et al., 2019; Karpukhin et al., 2020; Guu et al., 2020; Lewis et al., 2020; Xiong et al., 2021; Balachandran et al., 2021). The reader model then encodes all the passages through a transformer encoder separately, as trans- formers have quadratic computation with input se- quence length. Extractive span-based methods (Lee et al., 2019; Karpukhin et al., 2020; Guu et al., 2020; Lewis et al., 2020; Xiong et al., 2021) use these encoded representations for a per-passage se- lection of the answer span. Generative models such as FiD(Izacard and Grave, 2021b), and derivative models utilizing a similar reader, such as Izacard and Grave (2021a); Sachan et al. (2021); Singh et al. (2021); Lee et al. (2021) use their decoder to attend to all the passages simultaneously by concatenat- ing their encoded representations. The encoder- decoder cross attention enables the generative read- ers to fuse information globally across different passages. So far, the existing works have tried to perform global information fusion by adding a decoder and hence adopting the generative approach. We hy- pothesize that this is not efficient for extractive tasks where the answer exists in the provided con- …

Content truncated to 500 words