What Kind of Language Is Hard to Language-Model?

What Kind of Language Is Hard to Language-Model? Sabrina J. Mielke1 Ryan Cotterell1 Kyle Gorman2,3 Brian Roark3 Jason Eisner1 1 Department of Computer Science, Johns Hopkins University 2 Program in Linguistics, Graduate Center, City University of New York 3 Google {sjmielke@,ryan.cotterell@}jhu.edu kgorman@gc.cuny.edu roark@google.com jason@cs.jhu.edu Abstract How language-agnostic are current state-of- the-art NLP tools? Are there some types of language that are easier to model with cur- rent methods? In prior work (Cotterell et al., 2018) we attempted to address this question for language modeling, and observed that re- current neural network language models do not perform equally well over all the high- resource European languages found in the Europarl corpus. We speculated that inflec- tional morphology may be the primary cul- prit for the discrepancy. In this paper, we ex- tend these earlier experiments to cover 69 lan- guages from 13 language families using a mul- tilingual Bible corpus. Methodologically, we introduce a new paired-sample multiplicative mixed-effects model to obtain language dif- ficulty coefficients from at-least-pairwise par- allel corpora. In other words, the model is aware of inter-sentence variation and can han- dle missing data. Exploiting this model, we show that “translationese” is not any easier to model than natively written language in a fair comparison. Trying to answer the question of what features difficult languages have in com- mon, we try and fail to reproduce our earlier (Cotterell et al., 2018) observation about mor- phological complexity and instead reveal far simpler statistics of the data that seem to drive complexity in a much larger sample. 1 Introduction Do current NLP tools serve all languages? Tech- nically, yes, as there are rarely hard constraints that prohibit application to specific languages, as long as there is data annotated for the task. However, in practice, the answer is more nuanced: as most studies seem to (unfairly) assume English is representative of the world’s languages (Bender, 2009), we do not have a clear idea how well models perform cross-linguistically in a controlled setting. In this work, we look at current methods for lan- guage modeling and attempt to determine whether Resump- tion of the session Wieder- aufnah- me der ... The peace that ... Der gestern verein- ... Мирът, който беше ... Obwohl wir nicht ... Макар че не бяхме ... Now we can fi- nally ... Накрая всички можем ... en de bg 1 2 3 4 aligned multi-text language model 푦1,en 푦1,de 푦2,en 푦2,de 푦2,bg 푦3,de 푦3,bg 푦4,en 푦4,bg Difficulty estimation from sentence surprisal 푛1 푛2 푛3 푛4 ⇒ ⇒ ⇒ ⇒ 푑en 푑de 푑bg ⇒ ⇒ ⇒ Figure 1: Jointly estimating the information 푛푖 present in each multi-text intent 푖 and the difficulty 푑 푗 of each language 푗. At left, gray text indicates translations of the original (white) sentence in the same row. At right, darker cells indicate higher surprisal/difficulty. Empty cells indicate missing translations. English (en) is miss- ing a hard sentence and Bulgarian (bg) is missing an easy sentence, but this does not mislead our method into …

Content truncated to 500 words