Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1492–1502 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1137 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1492–1502 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1137 Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling Kazuya Kawakami♠ Chris Dyer♣ Phil Blunsom♠♣ ♠Department of Computer Science, University of Oxford, Oxford, UK ♣DeepMind, London, UK {kazuya.kawakami,phil.blunsom}@cs.ox.ac.uk,cdyer@google.com Abstract Fixed-vocabulary language models fail to account for one of the most characteristic statistical facts of natural language: the fre- quent creation and reuse of new word types. Although character-level language models offer a partial solution in that they can cre- ate word types not attested in the training corpus, they do not capture the “bursty” dis- tribution of such words. In this paper, we augment a hierarchical LSTM language model that generates sequences of word to- kens character by character with a caching mechanism that learns to reuse previously generated words. To validate our model we construct a new open-vocabulary lan- guage modeling corpus (the Multilingual Wikipedia Corpus; MWC) from compara- ble Wikipedia articles in 7 typologically diverse languages and demonstrate the ef- fectiveness of our model across this range of languages. 1 Introduction Language modeling is an important problem in nat- ural language processing with many practical ap- plications (translation, speech recognition, spelling autocorrection, etc.). Recent advances in neural networks provide strong representational power to language models with distributed representa- tions and unbounded dependencies based on recur- rent networks (RNNs). However, most language models operate by generating words by sampling from a closed vocabulary which is composed of the most frequent words in a corpus. Rare tokens are typically replaced by a special token, called the unknown word token, ⟨UNK⟩. Although fixed- vocabulary language models have some important practical applications and are appealing models for study, they fail to capture two empirical facts about the distribution of words in natural languages. First, vocabularies keep growing as the number of documents in a corpus grows: new words are con- stantly being created (Heaps, 1978). Second, rare and newly created words often occur in “bursts”, i.e., once a new or rare word has been used once in a document, it is often repeated (Church and Gale, 1995; Church, 2000). The open-vocabulary problem can be solved by dispensing with word-level models in favor of models that predict sentences as sequences of characters (Sutskever et al., 2011; Chung et al., 2017). Character-based models are quite success- ful at learning what (new) word forms look like (e.g., they learn a language’s orthographic conven- tions that tell us that sustinated is a plausible En- glish word and bzoxqir is not) and, when based on models that learn long-range dependencies such as RNNs, they can also be good models of how words fit together to form sentences. However, existing character-sequence models have no explicit mechanism for modeling the fact …

Content truncated to 500 words