What’s in a Name? Answer Equivalence For Open-Domain Question Answering

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9623–9629 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9623 What’s in a Name? Answer Equivalence For Open-Domain Question Answering Chenglei Si Computer Science University of Maryland clsi@terpmail.umd.edu Chen Zhao Computer Science University of Maryland chenz@cs.umd.edu Jordan Boyd-Graber CS, LSC, UMIACS, and iSchool University of Maryland jbg@umiacs.umd.edu Abstract A flaw in QA evaluation is that annotations of- ten only provide one gold answer. Thus, model predictions semantically equivalent to the an- swer but superficially different are considered incorrect. This work explores mining alias en- tities from knowledge bases and using them as additional gold answers (i.e., equivalent an- swers). We incorporate answers for two set- tings: evaluation with additional answers and model training with equivalent answers. We analyse three QA benchmarks: Natural Ques- tions, TriviaQA and SQuAD. Answer expansion increases the exact match score on all datasets for evaluation, while incorporating it helps model training over real-world datasets. We ensure the additional answers are valid through a human post hoc evaluation.1 1 Introduction: A Name that is the Enemy of Accuracy In question answering (QA), computers—given a question—provide the correct answer to the question. However, the modern formulation of QA usually assumes that each question has only one answer, e.g., SQuAD (Rajpurkar et al., 2016), HotpotQA (Yang et al., 2018), DROP (Dua et al., 2019). This is often a byproduct of the prevail- ing framework for modern QA (Chen et al., 2017; Karpukhin et al., 2020): a retriever finds passages that may contain the answer, and then a machine reader identifies the (as in only) answer span. In a recent position paper, Boyd-Graber and Börschinger (2020) argue that this is at odds with the best practices for human QA. This is also a prob- lem for computer QA. A BERT model (Devlin et al., 2019) trained on Natural Questions (Kwiatkowski et al., 2019, NQ) answers Tim Cook to the ques- tion “Who is the Chief Executive Officer of Apple?” (Figure 1), while the gold answer is only Timothy 1Our code and data are available at: https://github. com/NoviScl/AnswerEquiv. Question: Who is the Chief Executive Officer of Apple? Answer: Timothy Donald Cook Equivalent Answers: Tim Cook Passage: Tim Cook Timothy Donald Cook ( born November 1, 1960 ) is an American business executive and industrial engineer. Tim Cook is the chief executive officer of Apple inc. Figure 1: An example from NQ dataset. The correct model (BERT) prediction does not match the gold an- swer but matches the equivalent answer we mine from a knowledge base. Donald Cook, rendering Tim Cook as wrong as Tim Apple. In the 2020 NeurIPS Efficient QA com- petition (Min et al., 2021), human annotators rate nearly a third of the predictions that do not match the gold annotation as “definitely correct” or “pos- sibly correct”. Despite the near-universal acknowledgement of this problem, there is neither a clear measurement of its magnitude nor a consistent best practice solu- tion. While some datasets provide comprehensive …

Content truncated to 500 words