Two-Step Question Retrieval for Open-Domain QA

Findings of the Association for Computational Linguistics: ACL 2022, pages 1487 - 1492 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Two-Step Question Retrieval for Open-Domain QA Yeon Seonwoo†*, Juhee Son†*, Jiho Jin†, Sang-Woo Lee‡§, Ji-Hoon Kim‡§, Jung-Woo Ha‡§, Alice Oh† †KAIST, ‡NAVER AI Lab, §NAVER CLOVA {yeon.seonwoo,sjh5665,jinjh0123}@kaist.ac.kr {sang.woo.lee,genesis.kim,jungwoo.ha}@navercorp.com alice.oh@kaist.edu Abstract The retriever-reader pipeline has shown promis- ing performance in open-domain QA but suf- fers from a very slow inference speed. Recently proposed question retrieval models tackle this problem by indexing question-answer pairs and searching for similar questions. These mod- els have shown a significant increase in infer- ence speed, but at the cost of lower QA perfor- mance compared to the retriever-reader mod- els. This paper proposes a two-step question retrieval model, SQuID (Sequential Question- Indexed Dense retrieval) and distant supervi- sion for training. SQuID uses two bi-encoders for question retrieval. The first-step retriever selects top-k similar questions, and the second- step retriever finds the most similar question from the top-k questions. We evaluate the per- formance and the computational efficiency of SQuID. The results show that SQuID signif- icantly increases the performance of existing question retrieval models with a negligible loss on inference speed.1 1 Introduction Retriever-reader models in open-domain QA re- quire a long time for inference (Izacard and Grave, 2021; Lewis et al., 2020b; Sachan et al., 2021; Mao et al., 2021a; Karpukhin et al., 2020). This has been identified as a bottleneck in building real-time QA systems, and question retrieval and phrase-indexed QA have been proposed to resolve this problem (Seo et al., 2018, 2019; Lee et al., 2020, 2021a,b; Lewis et al., 2021a,b). These approaches directly search the answer of the input question from the corpus without conducting additional machine read- ing steps which are computationally inefficient. In phrase-indexed QA, retrievers pre-index all phrases in the corpus and find the most similar phrase to the input question. In question retrieval, synthetic *These authors contributed equally. 1The implementation of SQuID has been released at https://github.com/yeonsw/SQuID.git 0 200 400 600 800 1000 1200 1400 Inference Speed (questions/sec) 40 41 42 43 44 45 46 47 EM RePAQ-xlarge+Reranker-xxlarge RePAQ-large+Reranker-xlarge RePAQ-base+Reranker-base RePAQ-xlarge RePAQ-large RePAQ-base256 SQUID-RePAQ/BERT-large SQUID-RePAQ/BERT-base SQUID-RePAQ/DPR SQUID-RePAQ/DPR-parallel SQUID-RePAQ/ALBERT-large SQUID-RePAQ/ALBERT-base Figure 1: Trade-off relation between the open-domain QA performance and the inference time of existing ques- tion retrieval models (blue dots) and SQuID (red dots) on NaturalQuestions (NQ). The x-axis represents the inference speed and the y-axis represents the QA perfor- mance. question-answer pairs are pre-indexed and refer- enced by retrievers (Du et al., 2017; Duan et al., 2017; Fabbri et al., 2020; Lewis et al., 2020a). Although recent question retrieval models sig- nificantly increase the inference speed, this im- provement accompanies QA performance degra- dation. Several approaches have been applied to question retrieval models to overcome the perfor- mance degradation, such as adopting the cross- encoder (Mao et al., 2021b; Xiong et al., 2020) for re-ranking and increasing the model size (Lewis et al., 2021b). However, these approaches cause a significant loss of computational efficiency. Figure 1 …

Content truncated to 500 words