Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networks

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8519–8526, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8519 Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networks Shubham Toshniwal1, Sam Wiseman1, Allyson Ettinger2, Karen Livescu1, Kevin Gimpel1 1Toyota Technological Institute at Chicago 2Department of Linguistics, University of Chicago {shtoshni, swiseman, klivescu, kgimpel}@ttic.edu, aettinger@uchicago.edu Abstract Long document coreference resolution re- mains a challenging task due to the large mem- ory and runtime requirements of current mod- els. Recent work doing incremental corefer- ence resolution using just the global repre- sentation of entities shows practical benefits but requires keeping all entities in memory, which can be impractical for long documents. We argue that keeping all entities in memory is unnecessary, and we propose a memory- augmented neural network that tracks only a small bounded number of entities at a time, thus guaranteeing a linear runtime in length of document. We show that (a) the model remains competitive with models with high memory and computational requirements on OntoNotes and LitBank, and (b) the model learns an effi- cient memory management strategy easily out- performing a rule-based strategy. 1 Introduction Long document coreference resolution poses run- time and memory challenges. Current best models for coreference resolution have large memory re- quirements and quadratic runtime in the document length (Joshi et al., 2019; Wu et al., 2020), making them impractical for long documents. Recent work revisiting the entity-mention paradigm (Luo et al., 2004; Webster and Curran, 2014), which seeks to maintain explicit represen- tations only of entities, rather than all their con- stituent mentions, has shown practical benefits for memory while being competitive with state-of-the- art models (Xia et al., 2020). In particular, unlike other approaches to coreference resolution which maintain representations of both mentions and their corresponding entity clusters (Rahman and Ng, 2011; Stoyanov and Eisner, 2012; Clark and Man- ning, 2015; Wiseman et al., 2016; Lee et al., 2018) , the entity-mention paradigm stores representations only of the entity clusters, which are updated in- crementally as coreference predictions are made. While such an approach requires less memory than those that additionally store mention representa- tions, the number of entities can still become im- practically large when processing long documents, making the storing of all entity representations problematic. Is it necessary to maintain an unbounded number of mentions or entities? Psycholinguistic evidence suggests it is not, as human language processing is incremental (Tanenhaus et al., 1995; Keller, 2010) and has limited working memory (Baddeley, 1986). In practice, we find that most entities have a small spread (number of tokens from first to last mention of an entity), and thus do not need to be kept per- sistently in memory. This observation suggests that tracking a limited, small number of entities at any time can resolve the computational issues, albeit at a potential accuracy tradeoff. Previous work on finite memory models for coreference resolution has shown potential, but has been tested only on short documents (Liu et al., …

Content truncated to 500 words