Dynamic Entity Representations in Neural Language Models

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1830–1839 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Dynamic Entity Representations in Neural Language Models Yangfeng Ji∗ Chenhao Tan∗ Sebastian Martschat† Yejin Choi∗ Noah A. Smith∗ ∗Paul G. Allen School of Computer Science & Engineering, University of Washington †Department of Computational Linguistics, Heidelberg University {yangfeng,chenhao,yejin,nasmith}@cs.washington.edu martschat@cl.uni-heidelberg.de Abstract Understanding a long document requires tracking how entities are introduced and evolve over time. We present a new type of language model, ENTITYNLM, that can explicitly model entities, dynamically up- date their representations, and contextu- ally generate their mentions. Our model is generative and flexible; it can model an ar- bitrary number of entities in context while generating each entity mention at an arbi- trary length. In addition, it can be used for several different tasks such as language modeling, coreference resolution, and en- tity prediction. Experimental results with all these tasks demonstrate that our model consistently outperforms strong baselines and prior work. 1 Introduction Understanding a narrative requires keeping track of its participants over a long-term context. As a story unfolds, the information a reader associates with each character in a story increases, and ex- pectations about what will happen next change ac- cordingly. At present, models of natural language do not explicitly track entities; indeed, in today’s language models, entities are no more than the words used to mention them. In this paper, we endow a generative language model with the ability to build up a dynamic rep- resentation of each entity mentioned in the text. Our language model defines a probability distribu- tion over the whole text, with a distinct generative story for entity mentions. It explicitly groups those mentions that corefer and associates with each en- tity a continuous representation that is updated by every contextualized mention of the entity, and that in turn affects the text that follows. [John]1 wanted to go to [the coffee shop]2 in [downtown Copenhagen]3. [He]1 was told that [it]2 sold [the best beans]4. Figure 1: ENTITYNLM explicitly tracks entities in a text, including coreferring relationships be- tween entities like [John]1 and [He]1. As a lan- guage model, it is designed to predict that a coref- erent of [the coffee shop]2 is likely to follow “told that,” that the referring expression will be “it”, and that “sold the best beans” is likely to come next, by using entity information encoded in the dynamic distributed representation. Our method builds on recent advances in repre- sentation learning, creating local probability dis- tributions from neural networks. It can be un- derstood as a recurrent neural network language model, augmented with random variables for en- tity mentions that capture coreference, and with dynamic representations of entities. We estimate the model’s parameters from data that is annotated with entity mentions and coreference. Because our model is generative, it can be queried in different ways. Marginalizing every- thing except the words, it can play the role of a lan- guage model. In §5.1, we …

Content truncated to 500 words