Incremental Neural Coreference Resolution in Constant Memory

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8617–8624, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8617 Incremental Neural Coreference Resolution in Constant Memory Patrick Xia1 Jo˜ao Sedoc2 Benjamin Van Durme1 paxia@cs.jhu.edu jsedoc@nyu.edu vandurme@cs.jhu.edu 1Johns Hopkins University 2New York University Abstract We investigate modeling coreference resolu- tion under a fixed memory constraint by ex- tending an incremental clustering algorithm to utilize contextualized encoders and neural components. Given a new sentence, our end- to-end algorithm proposes and scores each mention span against explicit entity represen- tations created from the earlier document con- text (if any). These spans are then used to update the entity’s representations before be- ing forgotten; we only retain a fixed set of salient entities throughout the document. In this work, we successfully convert a high- performing model (Joshi et al., 2020), asymp- totically reducing its memory usage to con- stant space with only a 0.3% relative loss in F1 on OntoNotes 5.0. 1 Introduction Coreference resolution is a core task in NLP for both model analysis and information extraction. At the sentence level, ambiguities in pronoun coref- erence can be used to probe a model for com- mon sense (Levesque et al., 2012; Sakaguchi et al., 2020) or gender biases (Rudinger et al., 2018; Zhao et al., 2018). At the document level, coreference resolution is commonly used in information ex- traction pipelines, but can be applied to reading comprehension (Dasigi et al., 2019) or literature analysis (Bamman et al., 2014). Models for this task typically encode the en- tire text before scoring and subsequently clustering candidate mention spans, either found by a parser (Clark and Manning, 2016b) or learned jointly (Lee et al., 2017). Prior work has primarily focused on improving pairwise span scoring functions (Raghu- nathan et al., 2010; Clark and Manning, 2016a; Wu et al., 2020) and methods for decoding into globally consistent clusters (Wiseman et al., 2016; Lee et al., 2018; Kantor and Globerson, 2019; Xu and Choi, 2020). Recent models have also benefited from pre- trained encoders used to create high-dimensional input text (and span) representations, and improve- ments in contextualized encoders appear to trans- late directly to coreference resolution (Lee et al., 2018; Joshi et al., 2019, 2020). These models typically rely on simultaneous ac- cess to all spans – Θ(n) for a document with length n – for scoring and all scores – up to Θ(n2) – for decoding. As the dimensionality of contextualized encoders, and therefore the size of span represen- tations, increases, this becomes computationally intractable for long documents or under limited memory. Given these constraints, expensive scor- ing functions are increasingly difficult to explore. Further, prior models depart from how humans in- crementally read and reason about coreferent men- tions; Webster and Curran (2014) argue in favor of a limited memory constraint as a more psycholin- guistically plausible approach to reading and model coreference resolution via shift-reduce parsing. Motivated by scalability and armed with ad- vances in neural architectures, we revisit that …

Content truncated to 500 words