Using Morphological Knowledge in Open-Vocabulary Neural Language Models

Proceedings of NAACL-HLT 2018, pages 1435–1445 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Using Morphological Knowledge in Open-Vocabulary Neural Language Models Austin Matthews and Graham Neubig Language Technologies Institute Carnegie Mellon University Pittsburgh, PA, USA {austinma,gneubig}@cs.cmu.edu Chris Dyer DeepMind London, UK cdyer@google.com Abstract Languages with productive morphology pose problems for language models that generate words from a fixed vocabulary. Although character-based models allow any possible word type to be generated, they are linguis- tically naïve: they must discover that words exist and are delimited by spaces—basic lin- guistic facts that are built in to the structure of word-based models. We introduce an open- vocabulary language model that incorporates more sophisticated linguistic knowledge by predicting words using a mixture of three gen- erative processes: (1) by generating words as a sequence of characters, (2) by directly gener- ating full word forms, and (3) by generating words as a sequence of morphemes that are combined using a hand-written morphological analyzer. Experiments on Finnish, Turkish, and Russian show that our model outperforms character sequence models and other strong baselines on intrinsic and extrinsic measures. Furthermore, we show that our model learns to exploit morphological knowledge encoded in the analyzer, and, as a byproduct, it can perform effective unsupervised morphological disambiguation. 1 Introduction Language modelling of morphologically rich lan- guages is particularly challenging due to the vast set of potential word forms and the sparsity with which they appear in corpora. Traditional closed vocabulary models are unable to produce word forms unseen in training data and unable to gen- eralize sub-word patterns found in data. The most straightforward solution is to treat language as a sequence of characters (Sutskever et al., 2011). However, models that operate at two levels—a character level and a word level— have better performance (Chung et al., 2016). An- other solution is to use morphological information, which has shown benefits in non-neural models (Chahuneau et al., 2013). In this paper, we present a model that combines these approaches in a fully neural framework. Our model incorporates explicit morphological knowledge (e.g. from a finite-state morphological analyzer/generator) into a neural language model, combining it with existing word- and character- level modelling techniques, in order to create a model capable of successfully modelling morpho- logically complex languages. In particular, our model achieves three desirable properties. First, it conditions on all available (intra- sentential) context, allowing it, in principle, to capture long-range dependencies, such as that the verb agreement between “students” and “are” in the sentence “The students who studied the hardest are getting the highest grades”. While traditional n-gram based language models lack this property, RNN-based language models fulfill it. Second, it explicitly captures morphological variation, allowing sharing of information be- tween variants of the same word. This allows faster, smoother training as well as improved pre- dictive generalization. For example, if the model sees the phrase “gorped the ball” in data, it is able to infer that “gorping the ball” is also likely to be valid. …

Content truncated to 500 words