Modeling the Unigram Distribution

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3721–3729 August 1–6, 2021. ©2021 Association for Computational Linguistics 3721 Modeling the Unigram Distribution Irene Nikkarinen∗,D,7 Tiago Pimentel∗,D Dami´an E. Blasi@,9,6 Ryan CotterellD,Q DUniversity of Cambridge 7Yle @Harvard University 9MPI for Evolutionary Anthropology 6HSE University QETH Z¨urich irene.nikkarinen@gmail.com, tp472@cam.ac.uk dblasi@fas.harvard.edu, ryan.cotterell@inf.ethz.ch Abstract The unigram distribution is the non-contextual probability of finding a specific word form in a corpus. While of central importance to the study of language, it is commonly approxi- mated by each word’s sample frequency in the corpus. This approach, being highly depen- dent on sample size, assigns zero probability to any out-of-vocabulary (oov) word form. As a result, it produces negatively biased probabil- ities for any oov word form, while positively biased probabilities to in-corpus words. In this work, we argue in favor of properly modeling the unigram distribution—claiming it should be a central task in natural language process- ing. With this in mind, we present a novel model for estimating it in a language (a neural- ization of Goldwater et al.’s (2011) model) and show it produces much better estimates across a diverse set of 7 languages than the na¨ıve use of neural character-level language models. 1 Introduction Neural networks have yielded impressive gains in sentence-level language modeling across a typologically diverse set of languages (Mikolov et al., 2010; Kalchbrenner et al., 2016; Merity et al., 2018; Melis et al., 2018; Cotterell et al., 2018). Similarly, neural networks constitute the state of the art in modeling the distribution over a language’s word types (Pimentel et al., 2020), outperforming non-neural generative models such as Futrell et al.’s (2017) with character-level mod- els. This paper focuses on a less-researched task that is halfway between sentence-level language modeling and word type distributions: Modeling the unigram distribution, the distribution over word tokens in a language consisting of the prob- ability of a word’s form as well as its frequency in the language. In particular, as opposed to sentence-level modeling, the unigram distribution does not consider contextual information. ∗Equal contribution Figure 1: Word-level surprisal in Finnish under our two-stage model, two baseline LSTMs trained with either word type or token data, and another LSTM called the generator, trained on an interpolation of both. Lines depict rolling averages. The unigram distribution is a central object in the science of language from historical linguistics to psycholinguistics and beyond (Baayen et al., 2016; Diessel, 2017; Divjak, 2019). However, the major- ity of research on unigram distributions is based on identifying this distribution with sample fre- quency. This approach results in poor estimates, as it assigns zero probability to out-of-vocabulary words.1 Further, it is highly dependent on sample size (Baayen, 2002) The core contribution of our work is motivating the unigram distribution as a worthwhile objective for scientific inquiry—one which is currently un- derstudied in the field. With that in mind, we also present a neuralization of Goldwater et al.’s (2011) two-stage model.2 The gist of this approach is using two components to model the …

Content truncated to 500 words