Knowledge-Augmented Language Model and Its Application to Unsupervised Named-Entity Recognition

Proceedings of NAACL-HLT 2019, pages 1142–1150 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1142 Knowledge-Augmented Language Model and Its Application to Unsupervised Named-Entity Recognition Angli Liu Facebook AI anglil@cs.washington.edu Jingfei Du Facebook AI jingfeidu@fb.com Veselin Stoyanov Facebook AI ves@fb.com Abstract Traditional language models are unable to ef- ficiently model entity names observed in text. All but the most popular named entities appear infrequently in text providing insufficient con- text. Recent efforts have recognized that con- text can be generalized between entity names that share the same type (e.g., person or loca- tion) and have equipped language models with access to an external knowledge base (KB). Our Knowledge-Augmented Language Model (KALM) continues this line of work by aug- menting a traditional model with a KB. Un- like previous methods, however, we train with an end-to-end predictive objective optimizing the perplexity of text. We do not require any additional information such as named en- tity tags. In addition to improving language modeling performance, KALM learns to rec- ognize named entities in an entirely unsuper- vised way by using entity type information la- tent in the model. On a Named Entity Recog- nition (NER) task, KALM achieves perfor- mance comparable with state-of-the-art super- vised models. Our work demonstrates that named entities (and possibly other types of world knowledge) can be modeled success- fully using predictive learning and training on large corpora of text without any additional in- formation. 1 Introduction Language modeling is a form of unsupervised learning that allows language properties to be learned from large amounts of unlabeled text. As components, language models are useful for many Natural Language Processing (NLP) tasks such as generation (Parvez et al., 2018) and machine trans- lation (Bahdanau et al., 2014). Additionally, the form of predictive learning that language model- ing uses is useful to acquire text representations that can be used successfully to improve a number of downstream NLP tasks (Peters et al., 2018; De- vlin et al., 2018). In fact, models pre-trained with a predictive objective have provided a new state- of-the-art by a large margin. Current language models are unable to encode and decode factual knowledge such as the infor- mation about entities and their relations. Names of entities are an open class. While classes of named entities (e.g., person or location) occur fre- quently, each individual name (e.g, Atherton or Zhouzhuang) may be observed infrequently even in a very large corpus of text. As a result, language models learn to represent accurately only the most popular named entities. In the presence of external knowledge about named entities, language models should be able to learn to generalize across entity classes. For example, knowing that Alice is a name used to refer to a person should give ample infor- mation about the context in which the word may occur (e.g., Bob visited Alice). In this work, we propose Knowledge Aug- mented Language Model (KALM), a language model with access to information available in a KB. Unlike previous …

Content truncated to 500 words