Better Character Language Modeling through Morphology

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1606–1613 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1606 Better Character Language Modeling Through Morphology Terra Blevins1 and Luke Zettlemoyer1,2 1Paul G. Allen School of Computer Science & Engineering, University of Washington 2Facebook AI Research {blvns, lsz}@cs.washington.edu Abstract We incorporate morphological supervision into character language models (CLMs) via multitasking and show that this addition im- proves bits-per-character (BPC) performance across 24 languages, even when the morphol- ogy data and language modeling data are dis- joint. Analyzing the CLMs shows that in- flected words benefit more from explicitly modeling morphology than uninflected words, and that morphological supervision improves performance even as the amount of language modeling data grows. We then transfer mor- phological supervision across languages to im- prove language modeling performance in the low-resource setting. 1 Introduction Character language models (CLMs) are distri- butions over sequences of characters (Sutskever et al., 2011), in contrast to traditional language models which are distributions over sequences of words. CLMs eliminate the need for a fixed word vocabulary, and modeling text at the charac- ter level gives the CLM access to subword infor- mation. These attributes suggest that CLMs can model regularities that exist within words, such as morphological inflection. However, even large language modeling (LM) corpora have sparse cov- erage of inflected forms for morphologically-rich languages, which has been shown to make word and character language modeling more difficult (Gerz et al., 2018b; Cotterell et al., 2018). Be- cause to this, we hypothesize that accurately mod- eling morphology improves language modeling, but that it is difficult for CLMs to learn this from text alone. Motivated by this hypothesis, we add morphol- ogy supervision to character language modeling and show that, across two benchmark datasets, multitasking morphology with CLMs improves bits-per-character (BPC) performance on twenty- four languages, even when the annotated morphol- ogy features and language modeling data do not overlap. We also show that models augmented by multitasking achieve better BPC improvements on inflected forms than on uninflected forms, and that increasing the amount of language modeling data does not diminish the gains from morphol- ogy. Furthermore, to augment morphology anno- tations in low-resource languages, we also transfer morphology information between pairs of high- and low-resource languages. In this cross-lingual setting, we see that morphology supervision from the high-resource language improves BPC perfor- mance on the low-resource language over both the low-resource multitask model and over adding language modeling data from the high-resource language alone. 2 Approach Language Modeling Given a sequence of char- acters c = c1, c2, ..., cn, our character-level lan- guage models calculate the probability of c as p(c) = |c| � i=1 p(ci|c1, c2, ..., ci−1) (1) Each distribution is an LSTM (Hochreiter and Schmidhuber, 1997) trained such that at each time step t, the model takes in a character ct and es- timates the probability of the next character ct+1 as p(ct+1|c≤t) = g(LSTM(wt, ht−1)) (2) where ht−1 …

Content truncated to 500 words