Morphology Matters: A Multilingual Language Modeling Analysis

Morphology Matters: A Multilingual Language Modeling Analysis Hyunji Hayley Park University of Illinois hpark129@illinois.edu Katherine J. Zhang Carnegie Mellon University kjzhang@cmu.edu Coleman Haley Johns Hopkins University chaley7@jhu.edu Kenneth Steimel Indiana University ksteimel@iu.edu Han Liu University of Chicago∗ hanliu@uchicago.edu Lane Schwartz University of Illinois lanes@illinois.edu Abstract Prior studies in multilingual language model- ing (e.g., Cotterell et al., 2018; Mielke et al., 2019) disagree on whether or not inflectional morphology makes languages harder to model. We attempt to resolve the disagreement and extend those studies. We compile a larger corpus of 145 Bible translations in 92 lan- guages and a larger number of typological features.1 We fill in missing typological data for several languages and consider corpus- based measures of morphological complexity in addition to expert-produced typological features. We find that several morphological measures are significantly associated with higher surprisal when LSTM models are trained with BPE-segmented data. We also investigate linguistically motivated subword segmentation strategies like Morfessor and Finite-State Transducers (FSTs) and find that these segmentation strategies yield better performance and reduce the impact of a lan- guage’s morphology on language modeling. 1 Introduction With most research in Natural Language Pro- cessing (NLP) directed at a small subset of the world’s languages, whether the techniques devel- oped are truly language-agnostic is often not known. Because the vast majority of research fo- cuses on English, with Chinese a distant second (Mielke, 2016), neither of which is morphologi- cally rich, the impact of morphology on NLP tasks for various languages is not entirely understood. Several studies have investigated this issue in the context of language modeling by comparing a ∗Work done while at University of Colorado Boulder. 1https://github.com/hayleypark /MorphologyMatters. numberoflanguages, butfoundconflictingresults. Gerz et al. (2018) and Cotterell et al. (2018) find that morphological complexity is predictive of language modeling difficulty, while Mielke et al. (2019) conclude that simple statistics of a text like the number of types explain differences in modeling difficulty, rather than morphological measures. This paper revisits this issue by increasing the number of languages considered and augmenting the kind and number of morphological features used. We train language models for 92 languages from a corpus of Bibles fully aligned at the verse level and measure language modeling performance using surprisal (the negative log- likelihood) per verse (see §4.5). We investigate how this measure is correlated with 12 linguist- generated morphological features and four corpus- based measures of morphological complexity. Additionally, we contend that the relation be- tween segmentation method, morphology, and language modeling performance needs further investigation. Byte-Pair Encoding (BPE; Shibata et al., 1999) is widely used in NLP tasks including machine translation (Sennrich et al., 2016) as an unsupervised information-theoretic method for segmenting text data into subword units. Variants of BPE or closely related methods such as WordPiece (Kudo, 2018) are frequently used by state-of-the-art pretrained language models (Liu et al., 2019; Radford et al., 2019; Devlin et al., 2019; Yang et al., 2019). However, BPE and other segmentation methods may vary in how closely they capture morphological segments …

Content truncated to 500 words