RECONSIDER: Improved Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answering

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1280–1287 June 6–11, 2021. ©2021 Association for Computational Linguistics 1280 RECONSIDER: Improved Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answering Srinivasan Iyer† Sewon Min‡∗ Yashar Mehdad† Wen-tau Yih† †Facebook AI ‡University of Washington {sviyer,mehdad,scottyih}@fb.com sewon@cs.washington.edu Abstract State-of-the-art Machine Reading Comprehen- sion (MRC) models for Open-domain Ques- tion Answering (QA) are typically trained for span selection using distantly supervised posi- tive examples and heuristically retrieved neg- ative examples. This training scheme possi- bly explains empirical observations that these models achieve a high recall amongst their top few predictions, but a low overall accuracy, motivating the need for answer re-ranking. We develop a successful re-ranking approach (RECONSIDER) for span-extraction tasks that improves upon the performance of MRC mod- els, even beyond large-scale pre-training. RE- CONSIDER is trained on positive and nega- tive examples extracted from high confidence MRC model predictions, and uses in-passage span annotations to perform span-focused re- ranking over a smaller candidate set. As a re- sult, RECONSIDER learns to eliminate close false positives, achieving a new extractive state of the art on four QA tasks, with 45.5% Ex- act Match accuracy on Natural Questions with real user questions, and 61.7% on TriviaQA. We will release all related data, models, and code1. 1 Introduction Open-domain Question Answering (Voorhees et al., 1999) (QA) involves answering questions by ex- tracting correct answer spans from a large corpus of passages, and is typically accomplished by a light-weight passage retrieval model followed by a heavier Machine Reading Comprehension (MRC) model (Chen et al., 2017). The span selection com- ponents of MRC models are trained on distantly su- pervised positive examples (containing the answer string) together with heuristically chosen negative examples, typically from upstream retrieval models. This training scheme possibly explains empirical ∗Work done while at Facebook AI. 1github.com/facebookresearch/reconsider Figure 1: Top-3 passage-spans predicted by a BERT- MRC model on a question from NQ (answer spans are underlined). RECONSIDER re-evaluates the passages with marked candidate answers, eliminates close false positives and ranks Mikhail Gorbachev as correct. findings (Wang et al., 2018b,c) that while MRC models can confidently identify top-K answer can- didates (high recall), they cannot effectively dis- criminate between top semantically similar false positive candidates (low accuracy). In this paper, we develop a general approach to make answer re- ranking successful for span-extraction tasks, even over large pretrained models, and improve the state of the art on four QA datasets. Earlier work (Wang et al., 2018c,b) on open- domain QA have recognized the potential of answer re-ranking, which we continue to observe despite recent advances using large pre-trained models like BERT (Devlin et al., 2019). Figure 1 shows the top-3 predictions of a BERT-based SOTA model (Karpukhin et al., 2020) on a question from Nat- ural Questions (NQ) (Kwiatkowski et al., 2019), “Who was the head of the Soviet Union when it col- lapsed?" While all predictions are very relevant and refer to …

Content truncated to 500 words