Reference-Aware Language Models

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1850–1859 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Reference-Aware Language Models Zichao Yang1∗, Phil Blunsom2,3, Chris Dyer1,2, and Wang Ling2 1Carnegie Mellon University, 2DeepMind, and 3University of Oxford zichaoy@cs.cmu.edu, {pblunsom,cdyer,lingwang}@google.com Abstract We propose a general class of language models that treat reference as discrete stochastic latent variables. This decision allows for the creation of entity mentions by accessing external databases of refer- ents (required by, e.g., dialogue genera- tion) or past internal state (required to ex- plicitly model coreferentiality). Beyond simple copying, our coreference model can additionally refer to a referent using varied mention forms (e.g., a reference to “Jane” can be realized as “she”), a charac- teristic feature of reference in natural lan- guages. Experiments on three representa- tive applications show our model variants outperform models based on deterministic attention and standard language modeling baselines. 1 Introduction Referring expressions (REs) in natural language are noun phrases (proper nouns, common nouns, and pronouns) that identify objects, entities, and events in an environment. REs occur frequently and they play a key role in communicating infor- mation efficiently. While REs are common in nat- ural language, most previous work does not model them explicitly, either treating REs as ordinary words in the model or replacing them with special tokens that are filled in with a post processing step (Wen et al., 2015; Luong et al., 2015). Here we propose a language modeling framework that ex- plicitly incorporates reference decisions. In part, this is based on the principle of pointer networks in which copies are made from another source (G¨ulc¸ehre et al., 2016; Gu et al., 2016; Ling et al., ∗Work completed at DeepMind. dialogue recipe coref M: the nirala is a nice restuarant 1) soy milk 2) leaves 3) banana Blend soy milk and … [I]1 think…Go ahead [Linda]2 … thanks goes to [you]1 … a) reference to a list b) reference to a table c) reference to document context the nirala moderate lebanese ali baba moderate indian name price food Figure 1: Reference-aware language models. 2016; Vinyals et al., 2015; Ahn et al., 2016; Mer- ity et al., 2016). However, in the full version of our model, we go beyond simple copying and enable coreferent mentions to have different forms, a key characteristic of natural language reference. Figure 1 depicts examples of REs in the con- text of the three tasks that we consider in this work. First, many models need to refer to a list of items (Kiddon et al., 2016; Wen et al., 2015). In the task of recipe generation from a list of ingredients (Kiddon et al., 2016), the generation of the recipe will frequently refer to these items. As shown in Figure 1, in the recipe “Blend soy milk and ...”, soy milk refers to the ingredi- ent summaries. Second, reference to a database is crucial in many applications. One example is in task oriented dialogue where access to a database …

Content truncated to 500 words