UnitedQA: A Hybrid Approach for Open Domain Question Answering

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3080–3090 August 1–6, 2021. ©2021 Association for Computational Linguistics 3080 UnitedQA: A Hybrid Approach for Open Domain Question Answering Hao Cheng1∗, Yelong Shen2∗, Xiaodong Liu1, Pengcheng He2, Weizhu Chen2, Jianfeng Gao1 1 Microsoft Research 2 Microsoft Azure AI {chehao,yeshe,xiaodl,penhe,wzchen,jfgao}@microsoft.com Abstract To date, most of recent work under the retrieval-reader framework for open-domain QA focuses on either extractive or generative reader exclusively. In this paper, we study a hybrid approach for leveraging the strengths of both models. We apply novel techniques to enhance both extractive and generative readers built upon recent pretrained neural language models, and find that proper training meth- ods can provide large improvements over pre- vious state-of-the-art models. We demonstrate that an hybrid approach by combining answers from both readers can effectively take advan- tages of extractive and generative answer in- ference strategies and outperform single mod- els as well as homogeneous ensembles. Our approach outperforms previous state-of-the-art models by 3.3 and 2.7 points in exact match on NaturalQuestions and TriviaQA respectively. 1 Introduction Open-domain question answering (QA) has been a long standing problem in natural language under- standing, information retrieval, and related fields (Chen and Yih, 2020). An typical open-domain QA system follows the retrieval-reader framework (Chen et al., 2017; Guu et al., 2020; Karpukhin et al., 2020), where the relevant passages are first retrieved from a large text corpus, and a reader mod- ule then navigates multiple passages for answer inference. In this work, we study two paradigms of reader modules, i.e. extractive (Karpukhin et al., 2020; Guu et al., 2020) and generative (Lewis et al., 2020; Izacard and Grave, 2021) readers. The ex- tractive reader extracts contiguous spans from the retrieved passages whereas the generative reader sequentially decodes the answer string which might not be contained in the retrieved passages. ∗Equal Contribution Recent work on open-domain QA (Karpukhin et al., 2020; Guu et al., 2020; Lewis et al., 2020; Izacard and Grave, 2021) explores either an extrac- tive reader or a generative reader exclusively. We hypothesize that extractive and generative readers adopt different answer inference strategies, thus a hybrid extractive/generative reader can be a bet- ter option for open-domain QA tasks. As shown in Figure 1, compared with prediction agreement among only generative or extractive readers (top- left and bottom-right), the cross prediction agree- ment between extractive and generative readers (bottom-left) is relatively low (<50%). It indicates that answers produced by those two types of mod- els are different and they can be complementary to each other. Therefore, we propose a hybrid reader approach, UnitedQA, which is a simple ensemble approach to combine the predictions from extrac- tive and generative readers. It achieves state-of-the- art results on NaturalQuestions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017). In UnitedQA, the extractive reader (UnitedQA- E) and generative reader (UnitedQA-G) are built upon the pretrained language models, ELECTRA (Clark et al., 2020) and …

Content truncated to 500 words