Exploring Dual Encoder Architectures for Question Answering

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9414 - 9419 December 7-11, 2022 ©2022 Association for Computational Linguistics Exploring Dual Encoder Architectures for Question Answering Zhe Dong Jianmo Ni Dan Bikel Enrique Alfonseca Yuan Wang Chen Qu Imed Zitouni Google Inc {zhedong, jianmon, dbikel, ealfonseca, yuawang, cqu, izitouni}@google.com Abstract Dual encoders have been used for question- answering (QA) and information retrieval (IR) tasks with good results. Previous research fo- cuses on two major types of dual encoders, Siamese Dual Encoder (SDE), with parame- ters shared across two encoders, and Asym- metric Dual Encoder (ADE), with two dis- tinctly parameterized encoders. In this work, we explore different ways in which the dual encoder can be structured, and evaluate how these differences can affect their efficacy in terms of QA retrieval tasks. By evaluating on MS MARCO, open domain NQ and the Mul- tiReQA benchmarks, we show that SDE per- forms significantly better than ADE. We fur- ther propose three different improved versions of ADEs by sharing or freezing parts of the ar- chitectures between two encoder towers. We find that sharing parameters in projection lay- ers would enable ADEs to perform competi- tively with or outperform SDEs. We further explore and explain why parameter sharing in projection layer significantly improves the effi- cacy of the dual encoders, by directly probing the embedding spaces of the two encoder tow- ers with t-SNE algorithm (van der Maaten and Hinton, 2008). 1 Introduction A dual encoder is an architecture consisting of two encoders, each of which encodes an input (such as a piece of text) into an embedding, and where the model is optimized based on similarity metrics in the embedding space. It has shown an excellent performance in a wide range of information re- trieval and question answering tasks (Gillick et al., 2018; Karpukhin et al., 2020). This approach is also easy to productionize because the embedding index of dual encoders can grow dynamically for Code and additional information: https://sites.google.com/view/ explore-dual-encoder-architect. Model Architecture DPR (Karpukhin et al., 2020) Asymmetric DensePhrases (Lee et al., 2021a) Asymmetric SBERT (Reimers and Gurevych, 2019) Siamese ST5 (Ni et al., 2021b) Siamese Table 1: Existing off-the-shelf dual encoders. newly discovered or updated documents and pas- sages without retraining the encoders (Gillick et al., 2018). In contrast, generative neural networks used for question answering need to be retrained with new data. This advantage makes dual encoders more robust to freshness. There are different valid designs for dual en- coders. As shown in Table 1, previous research focuses on the two major types: Siamese Dual Encoder (SDE) and Asymmetric Dual Encoder (ADE). In a SDE, the parameters are shared be- tween the two encoders. In an ADE, only some or no parameters are shared (Gillick et al., 2018). In practice, we often require certain asymmetry in the dual encoders, especially in the case where the inputs of the two towers are of different types. Al- though all of these models have achieved excellent results …

Content truncated to 500 words