Figurative Language in Recognizing Textual Entailment

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 344–350 August 1–6, 2021. ©2021 Association for Computational Linguistics 344 Reader-Guided Passage Reranking for Open-Domain Question Answering Yuning Mao1∗, Pengcheng He2, Xiaodong Liu3, Yelong Shen2, Jianfeng Gao3, Jiawei Han1, Weizhu Chen2 1University of Illinois, Urbana-Champaign 2Microsoft Azure AI 3Microsoft Research 1{yuningm2, hanj}@illinois.edu 2,3{penhe, xiaodl, yeshe, jfgao,wzchen}@microsoft.com Abstract Current open-domain question answering sys- tems often follow a Retriever-Reader architec- ture, where the retriever first retrieves rele- vant passages and the reader then reads the retrieved passages to form an answer. In this paper, we propose a simple and effec- tive passage reranking method, named Reader- guIDEd Reranker (RIDER), which does not in- volve training and reranks the retrieved pas- sages solely based on the top predictions of the reader before reranking. We show that RIDER, despite its simplicity, achieves 10 to 20 abso- lute gains in top-1 retrieval accuracy and 1 to 4 Exact Match (EM) gains without refining the retriever or reader. In addition, RIDER, with- out any training, outperforms state-of-the-art transformer-based supervised rerankers. Re- markably, RIDER achieves 48.3 EM on the Natural Questions dataset and 66.4 EM on the TriviaQA dataset when only 1,024 tokens (7.8 passages on average) are used as the reader in- put after passage reranking.1 1 Introduction Current open-domain question answering (OpenQA) systems often follow a Retriever- Reader (R2) architecture, where the retriever first retrieves relevant passages and the reader then reads the retrieved passages to form an answer. Since the retriever retrieves passages from a large candidate pool (e.g., millions of Wikipedia passages), it often fails to rank the most relevant passages at the very top. One line of work (Mao et al., 2020; Karpukhin et al., 2020) aims to improve the retriever and shows that significantly better QA performance can be achieved when the retrieval results are improved. ∗Work was done during internship at Microsoft Azure AI. 1Our code is available at https://github.com/ morningmoni/GAR. An alternative solution is to rerank the initial re- trieval results via a reranker, which is widely used in information retrieval (Nogueira and Cho, 2019; Qiao et al., 2019) and explored in early OpenQA systems (Wang et al., 2018a; Lee et al., 2018). However, current state-of-the-art OpenQA systems (Karpukhin et al., 2020; Izacard and Grave, 2020b) do not distinguish the order of the retrieved pas- sages and instead equally consider a large number of retrieved passages (e.g., 100), which could be computationally prohibitive as the model size of the readers becomes larger (Izacard and Grave, 2020b). We argue that a Retriever-Reranker-Reader (R3) architecture is beneficial in terms of both model effectiveness and efficiency: passage reranking im- proves the retrieval accuracy of the retriever at top positions and allows the reader to achieve compara- ble performance with fewer passages as the input. However, one bottleneck of R3 is that the reranker, previously based on BiLSTM (Wang et al., 2018a; Lee et al., 2018) and nowadays typically BERT- based cross-encoder (Nogueira and Cho, 2019; Qiao et al., 2019), is often costly …

Content truncated to 500 words