A Neural Language Model for Dynamically Representing the Meanings of Unknown Words and Entities in a Discourse

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 473–483, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP A Neural Language Model for Dynamically Representing the Meanings of Unknown Words and Entities in a Discourse Sosuke Kobayashi Preferred Networks, Inc., Japan sosk@preferred.jp Naoaki Okazaki Tokyo Institute of Technology, Japan okazaki@c.titech.ac.jp Kentaro Inui Tohoku University / RIKEN, Japan inui@ecei.tohoku.ac.jp Abstract This study addresses the problem of iden- tifying the meaning of unknown words or entities in a discourse with respect to the word embedding approaches used in neural language models. We proposed a method for on-the-fly construction and exploitation of word embeddings in both the input and output layers of a neural model by tracking contexts. This extends the dynamic entity representation used in Kobayashi et al. (2016) and incorporates a copy mechanism proposed independently by Gu et al. (2016) and Gulcehre et al. (2016). In addition, we construct a new task and dataset called Anonymized Lan- guage Modeling for evaluating the abil- ity to capture word meanings while read- ing. Experiments conducted using our novel dataset show that the proposed vari- ant of RNN language model outperformed the baseline model. Furthermore, the ex- periments also demonstrate that dynamic updates of an output layer help a model predict reappearing entities, whereas those of an input layer are effective to predict words following reappearing entities. 1 Introduction Language models that use probability distribu- tions over sequences of words are found in many natural language processing applications, includ- ing speech recognition, machine translation, text summarization, and dialogue utterance genera- tion. Recent studies have demonstrated that lan- guage models trained using neural network (Ben- gio et al., 2003; Mikolov et al., 2010) such as re- current neural network (RNN) (Jozefowicz et al., 2016) and convolutional neural network (Dauphin ... [ 1 ] killed [ 2 ] with bombs … ... police suspects [ 1 ] attacked ... ... police will arrest [ 1 ] … ... will arrest [ 1 ] soon … d[2],2 d[1],2 ! ! ! ! x[1] x[2] y[1] y[2] = −−−→ RNN( Figure 1: Dynamic Neural Text Modeling: the embeddings of unknown words, denoted by coref- erence indexes “[ k ]” are dynamically computed and used in both the input and output layers (x[k] and y[k]) of a RNN language model. These are constructed from contextual information (d[k],i) preceding the current (i + 1)-th sentence. et al., 2016) achieve the best performance across a range of corpora (Mikolov et al., 2010; Chelba et al., 2014; Merity et al., 2017; Grave et al., 2017). However, current neural language models have a major drawback: the language model works only when applied to a closed vocabulary of fixed size (usually comprising high-frequency words from the given training corpus). All occurrences of out- of-vocabulary words are replaced with a single dummy token “<unk>”, showing that the word is unknown. For example, the word sequence, Piko- taro sings PPAP on YouTube is treated as <unk> sings <unk> on <unk> …

Content truncated to 500 words