Benchmarking Scalable Methods for Streaming Cross Document Entity Coreference

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4717–4731 August 1–6, 2021. ©2021 Association for Computational Linguistics 4717 Benchmarking Scalable Methods for Streaming Cross Document Entity Coreference Robert L. Logan IV∗♦ Andrew McCallum♠♥ Sameer Singh♦ Daniel Bikel♠ ♦University of California, Irvine ♠Google Research ♥University of Massachusetts, Amherst {rlogan, sameer}@uci.edu {mccallum, dbikel}@google.com Abstract Streaming cross document entity corefer- ence (CDC) systems disambiguate mentions of named entities in a scalable manner via in- cremental clustering. Unlike other approaches for named entity disambiguation (e.g., entity linking), streaming CDC allows for the disam- biguation of entities that are unknown at in- ference time. Thus, it is well-suited for pro- cessing streams of data where new entities are frequently introduced. Despite these benefits, this task is currently difficult to study, as exist- ing approaches are either evaluated on datasets that are no longer available, or omit other cru- cial details needed to ensure fair comparison. In this work, we address this issue by compil- ing a large benchmark adapted from existing free datasets, and performing a comprehensive evaluation of a number of novel and existing baseline models.1 We investigate: how to best encode mentions, which clustering algorithms are most effective for grouping mentions, how models transfer to different domains, and how bounding the number of mentions tracked dur- ing inference impacts performance. Our re- sults show that the relative performance of neu- ral and feature-based mention encoders varies across different domains, and in most cases the best performance is achieved using a combi- nation of both approaches. We also find that performance is minimally impacted by limit- ing the number of tracked mentions. 1 Introduction The ability to disambiguate mentions of named entities in text is a central task in the field of in- formation extraction, and is crucial to topic track- ing, knowledge base induction and question an- swering. Recent work on this problem has fo- cused almost solely on entity linking–based ap- ∗Work done during an internship at Google Research. 1Code and data available at: https://github.com/ rloganiv/streaming-cdc proaches, i.e., models that link mentions to a fixed set of known entities. While significant strides have been made on this front—with systems that can be trained end-to-end (Kolitsas et al., 2018), on millions of entities (Ling et al., 2020), and link to entities using only their textual descriptions (Lo- geswaran et al., 2019)—all entity linking systems suffer from the significant limitation that they are restricted to linking to a curated list of entities that is fixed at inference time. Thus they are of limited use when processing data streams where new enti- ties regularly appear, such as research publications, social media feeds, and news articles. In contrast, the alternative approach of cross- document entity coreference (CDC) (Bagga and Baldwin, 1998; Gooi and Allan, 2004; Singh et al., 2011; Dutta and Weikum, 2015), which disam- biguates mentions via clustering, does not suffer from this shortcoming. Instead most CDC algo- rithms suffer …

Content truncated to 500 words