The Referential Reader: A Recurrent Entity Network for Anaphora Resolution

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5918–5925 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5918 The Referential Reader: A Recurrent Entity Network for Anaphora Resolution Fei Liu ∗ The University of Melbourne Victoria, Australia Luke Zettlemoyer Facebook AI Research University of Washington Seattle, USA Jacob Eisenstein Facebook AI Research Seattle, USA Abstract We present a new architecture for storing and accessing entity mentions during online text processing. While reading the text, entity ref- erences are identified, and may be stored by either updating or overwriting a cell in a fixed- length memory. The update operation implies coreference with the other mentions that are stored in the same cell; the overwrite opera- tion causes these mentions to be forgotten. By encoding the memory operations as differen- tiable gates, it is possible to train the model end-to-end, using both a supervised anaphora resolution objective as well as a supplementary language modeling objective. Evaluation on a dataset of pronoun-name anaphora demon- strates strong performance with purely incre- mental text processing. 1 Introduction Reference resolution is fundamental to language understanding. Current state-of-the-art systems employ the mention-pair model, in which a clas- sifier is applied to all pairs of spans (e.g., Lee et al., 2017). This approach is expensive in both computation and labeled data, and it is also cogni- tively implausible: human readers interpret text in a nearly online fashion (Tanenhaus et al., 1995). We present a new method for reference resolu- tion, which reads the text left-to-right while stor- ing entities in a fixed-size working memory (Fig- ure 1). As each token is encountered, the reader must decide whether to: (a) link the token to an existing memory, thereby creating a coreference link, (b) overwrite an existing memory and store a new entity, or (c) disregard the token and move ahead. As memories are reused, their salience in- creases, making them less likely to be overwritten. This online model for coreference resolution is based on the memory network architecture (We- ∗Work carried out as an intern at Facebook AI Research M(1) M(2) Ismael told Captain Ahab he saw Moby-Dick o(1) 1 u(1) 5 o(2) 3 u(2) 4 o(2) 7 self link coreferential not coreferential  Figure 1: A referential reader with two memory cells. Overwrite and update are indicated by o(i) t and u(i) t ; in practice, these operations are continuous gates. Thick- ness and color intensity of edges between memory cells at neighboring steps indicate memory salience;  indi- cates an overwrite. ston et al., 2015), in which memory operations are differentiable, enabling end-to-end training from gold anaphora resolution data. Furthermore, the memory can be combined with a recurrent hidden state, enabling prediction of the next word. This makes it possible to train the model from unla- beled data using a language modeling objective. To summarize, we present a model that pro- cesses the text incrementally, resolving references on the fly (Schlangen et al., 2009). The model yields …

Content truncated to 500 words