Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level Prediction

Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level Prediction Daniela Gerz1, Ivan Vuli´c1, Edoardo Ponti1 Jason Naradowsky3, Roi Reichart2 Anna Korhonen1 1Language Technology Lab, DTAL, University of Cambridge 2Faculty of Industrial Engineering and Management, Technion, IIT 3Johns Hopkins University 1{dsg40,iv250,ep490,alk23}@cam.ac.uk 2roiri@ie.technion.ac.il 3narad@jhu.edu Abstract Neural architectures are prominent in the con- struction of language models (LMs). How- ever, word-level prediction is typically agnos- tic of subword-level information (characters and character sequences) and operates over a closed vocabulary, consisting of a limited word set. Indeed, while subword-aware mod- els boost performance across a variety of NLP tasks, previous work did not evaluate the abil- ity of these models to assist next-word predic- tion in language modeling tasks. Such subword- level informed models should be particularly effective for morphologically-rich languages (MRLs) that exhibit high type-to-token ratios. In this work, we present a large-scale LM study on 50 typologically diverse languages cover- ing a wide variety of morphological systems, and offer new LM benchmarks to the commu- nity, while considering subword-level informa- tion. The main technical contribution of our work is a novel method for injecting subword- level information into semantic word vectors, integrated into the neural language modeling training, to facilitate word-level prediction. We conduct experiments in the LM setting where the number of infrequent words is large, and demonstrate strong perplexity gains across our 50 languages, especially for morphologically- rich languages. Our code and data sets are pub- licly available. 1 Introduction Language Modeling (LM) is a key NLP task, serving as an important component for applications that re- quire some form of text generation, such as machine translation (Vaswani et al., 2013), speech recognition (Mikolov et al., 2010), dialogue generation (Serban et al., 2016), or summarisation (Filippova et al., 2015). A traditional recurrent neural network (RNN) LM setup operates on a limited closed vocabulary of words (Bengio et al., 2003; Mikolov et al., 2010). The limitation arises due to the model learning pa- rameters exclusive to single words. A standard train- ing procedure for neural LMs gradually modifies the parameters based on contextual/distributional infor- mation: each occurrence of a word token in train- ing data contributes to the estimate of a word vector (i.e., model parameters) assigned to this word type. Low-frequency words therefore often have incorrect estimates, not having moved far from their random initialisation. A common strategy for dealing with this issue is to simply exclude the low-quality param- eters from the model (i.e., to replace them with the <unk> placeholder), leading to only a subset of the vocabulary being represented by the model. This limited vocabulary assumption enables the model to bypass the problem of unreliable word es- timates for low-frequency and unseen words, but it does not resolve it. The assumption is far from ideal, partly due to the Zipfian nature of each language (Zipf, 1949), and its limitation is even more pro- nounced for morphologically-rich languages (MRLs): these languages inherently generate a plethora of words by their morphological systems. As a conse- quence, there …

Content truncated to 500 words