Scaling Within Document Coreference to Long Texts

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3921–3931 August 1–6, 2021. ©2021 Association for Computational Linguistics 3921 Scaling Within Document Coreference to Long Texts Raghuveer Thirukovalluru1, Nicholas Monath1, Kumar Shridhar2, Manzil Zaheer3, Mrinmaya Sachan2and Andrew McCallum1 1UMass Amherst, 2ETH Zurich, 3Google rthirukovall@umass.edu, {nmonath,mccallum}@cs.umass.edu {shridhar.kumar,mrinmaya.sachan}@inf.ethz.ch manzilzaheer@google.com Abstract State of the art end-to-end coreference reso- lution models use expensive span representa- tions and antecedent prediction mechanisms. These approaches are expensive both in terms of their memory requirements as well as com- pute time, and are particularly ill-suited for long documents. In this paper, we propose an approximation to end-to-end models which scales gracefully to documents of any length. Replacing span representations with token rep- resentations, we reduce the time/memory com- plexity via token windows and nearest neigh- bor sparsification methods for more efficient antecedent prediction. We show our ap- proach’s resulting reduction of training and inference time compared to state-of-the-art methods with only a minimal loss in accuracy. 1 Introduction Recent advances in coreference resolution (Lee et al., 2018; Joshi et al., 2019, 2020; Wu et al., 2020) have been largely based on the end-to-end model proposed by Lee et al. (2017). However, these models are costly both in terms of training and inference time, as well as memory require- ments, especially for long documents. The large computational cost makes the models infeasible to run for a typical user on large document collections in domains such as blogs, stories, books, etc. More- over, a reduction in energy use of these models can be of benefit to cloud service providers’ costs and there can also be environmental benefits (Strubell et al., 2019; Schwartz et al., 2020). There are two main computational bottlenecks in using end-to-end coreference models on long documents: (i) span and span-pair representations for all spans in the document are simultaneously considered, and (ii) the coreference decision for a mention requires considering all candidate an- tecedent spans. In this paper, we propose an approximation to the end-to-end coreference model (Lee et al., 2017) that scales to long documents by addressing both these bottlenecks. Our proposed approach operates at the token level instead of the span level, remov- ing the quadratic dependence on the number of mention spans in a document and addressing bottle- neck (i). We propose token level scoring functions for the bilinear inference model originally proposed by Lee et al. (2018). To address bottleneck (ii), we use token windows, token-level k-nearest neighbor relationships along with low-rank matrix approxi- mations of the token similarity matrix thereby im- proving time/memory efficiency. We also propose an approach to drop token representations from memory to reduce memory requirements while still maintaining the accuracy. We evaluate our approach on three coreference datasets: CoNLL-2012 (Pradhan et al., 2012), Lit- bank (Bamman et al., 2019), and MedMentions (Mohan and Li, 2019) and observe competitive ac- curacy to state of the art coreference models based on end-to-end training while achieving both faster training and inference running times. Our approach is also more memory efficient …

Content truncated to 500 words