Attending to Long-Distance Document Context for Sequence Labeling

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3692–3704 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3692 Attending to Long-Distance Document Context for Sequence Labeling Matthew J¨orke∗ Computer Science Department Stanford University joerke@stanford.edu Jon Gillick School of Information UC Berkeley jongillick@berkeley.edu Matthew Sims School of Information UC Berkeley mbsims@berkeley.edu David Bamman School of Information UC Berkeley dbamman@berkeley.edu Abstract We present in this work a method for in- corporating global context in long documents when making local decisions in sequence la- beling problems like NER. Inspired by work in featurized log-linear models (Chieu and Ng, 2002; Sutton and McCallum, 2004), our model learns to attend to multiple mentions of the same word type in generating a representa- tion for each token in context, extending that work to learning representations that can be incorporated into modern neural models. At- tending to broader context at test time pro- vides complementary information to pretrain- ing (Gururangan et al., 2020), yields strong gains over equivalently parameterized models lacking such context, and performs best at rec- ognizing entities with high TF-IDF scores (i.e., those that are important within a document). 1 Introduction Many of the main datasets used in NLP are com- prised of relatively short documents: English OntoNotes (Weischedel et al., 2012), for exam- ple, contains an average of 223 tokens per docu- ment, the WSJ portion of the Penn Treebank (Mar- cus et al., 1993) averages 501 tokens, the IMDb dataset (Maas et al., 2011) averages 272 tokens, and SQuAD 2.0 (Rajpurkar et al., 2018) contains an av- erage of 134 tokens per passage. This focus has, in turn, led to the development of models specifi- cally optimized for the characteristics of short doc- uments, including a pervasive focus on the sentence as the atomic unit of analysis for such tasks as NER and parsing, and influencing the maximum context ∗Work completed while at UC Berkeley. length of contextual language models like BERT (Devlin et al., 2019) to be limited to 512 tokens. At the same time, however, longer documents are increasingly the objects of empirical study in ar- eas as diverse as computational social science and the digital humanities—including novels (Piper, 2018; Underwood, 2019), scientific articles (Jur- gens et al., 2018) and political manifestos (Menini et al., 2017; Denny and Spirling, 2018). These long documents present not only challenges for NLP (such as any task, like coreference resolution, whose computational complexity is superlinear in the size of the document) but opportunities as well, since the longer document context presents greater opportunity for learning better representations. Recent work in NLP has begun exploring this link between longer documents and representa- tion learning. First, while contextualized mod- els (e.g. Peters et al., 2018; Devlin et al., 2019) generally consider the context of a few sentences, several recent advancements have enabled signif- icantly longer input sequences (e.g. Dai et al., 2019; Beltagy et al., 2020; Kitaev et al., 2020; Rae et al., 2019); most, however, are either incapable of processing book-level …

Content truncated to 500 words