Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLP

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4472–4485 August 1–6, 2021. ©2021 Association for Computational Linguistics 4472 Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLP Anthony Chen∗ Pallavi Gudipati Shayne Longpre Xiao Ling Sameer Singh University of California, Irvine Apple {anthony.chen, sameer}@uci.edu {pgudipati, slongpre, xiaoling}@apple.com Abstract Retrieval is a core component for open-domain NLP tasks. In open-domain tasks, multiple en- tities can share a name, making disambigua- tion an inherent yet under-explored problem. We propose an evaluation benchmark for as- sessing the entity disambiguation capabilities of these retrievers, which we call Ambiguous Entity Retrieval (AmbER) sets. We define an AmbER set as a collection of entities that share a name along with queries about those entities. By covering the set of entities for polysemous names, AmbER sets act as a challenging test of entity disambiguation. We create AmbER sets for three popular open-domain tasks: fact checking, slot filling, and question answering, and evaluate a diverse set of retrievers. We find that the retrievers exhibit popularity bias, sig- nificantly under-performing on rarer entities that share a name, e.g., they are twice as likely to retrieve erroneous documents on queries for the less popular entity under the same name. These experiments on AmbER sets show their utility as an evaluation tool and highlight the weaknesses of popular retrieval systems.1 1 Introduction Substantial progress in NLP has been made on “closed” tasks, where queries are paired with rele- vant documents (Rajpurkar et al., 2016; Dua et al., 2019). However, there is growing interest in “open- domain” tasks, where relevant documents need to be retrieved from a knowledge source before an NLP system can perform reasoning and pro- duce an answer (Chen et al., 2017; Petroni et al., 2021). The open-domain setting better reflects real-world usage for tasks where relevant informa- tion is generally not provided (e.g., fact checking). ∗Work started during an internship at Apple. 1The AmbER sets used in this paper and the code to generate them are available at https://github.com/ anthonywchen/AmbER-Sets. Q: Which battle did Abe Lincoln fight in? A: World War II Wikipedia Documents Ranked by BLINK: 1. Abraham Lincoln 2. Abraham Lincoln in the Black Hawk War 3. Abraham Lincoln (captain) 4. Benjamin Lincoln 5. Lincoln Nebraska 6. Lincoln England Q: What musical instrument does Abe Lincoln play? A: Trombone Wikipedia Documents Ranked by BLINK: 1. Abraham Lincoln 2. John Wilkes Booth 3. Abe (musical) 4. Nebraska 5. Lincoln Nebraska 6. Abe Lincoln (musician) Figure 1: Queries for two entities (president & mu- sician) with the name “Abe Lincoln”. Retrieving the gold document involves disambiguating which “Abe Lincoln” each query is asking about. BLINK performs sub-optimally on the second query, as it ranks the doc- ument of the president over the gold document. Because success hinges on finding relevant docu- ments, open-domain progress has been closely tied to improvements in retrieval systems2 (Lee et al., 2019; Karpukhin et al., 2020; …

Content truncated to 500 words