Show Some Love to Your n-grams: A Bit of Progress and Stronger n-gram Language Modeling Baselines

Proceedings of NAACL-HLT 2019, pages 4113–4118 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 4113 Show Some Love to Your 푛-grams: A Bit of Progress and Stronger 푛-gram Language Modeling Baselines Ehsan Shareghi, Daniela Gerz, Ivan Vulić, Anna Korhonen Language Technology Lab, DTAL, University of Cambridge {es776, dsg40, iv250, alk23}@cam.ac.uk Abstract In recent years neural language models (LMs) have set state-of-the-art performance for sev- eral benchmarking datasets. While the reasons for their success and their computational de- mand are well-documented, a comparison be- tween neural models and more recent devel- opments in 푛-gram models is neglected. In this paper, we examine the recent progress in 푛-gram literature, running experiments on 50 languages covering all morphological lan- guage families. Experimental results illustrate that a simple extension of Modified Kneser- Ney outperforms an LSTM language model on 42 languages while a word-level Bayesian 푛- gram LM (Shareghi et al., 2017) outperforms the character-aware neural model (Kim et al., 2016) on average across all languages, and its extension which explicitly injects linguis- tic knowledge (Gerz et al., 2018a) on 8 lan- guages. Further experiments on larger Eu- roparl datasets for 3 languages indicate that neural architectures are able to outperform computationally much cheaper 푛-gram models: 푛-gram training is up to 15, 000× quicker. Our experiments illustrate that standalone 푛-gram models lend themselves as natural choices for resource-lean or morphologically rich lan- guages, while the recent progress has signifi- cantly improved their accuracy. 1 Introduction Statistical language models (LMs) are the pivot for several natural language processing tasks where a model trained on a text corpus is required to assign a probability to a given sequence 푤1푤2...푤푁 (de- noted by 푤푁 1 ). This probability indicates how likely is for 푤푁 1 to belong to the corpus and is decomposed into conditional probabilities of words given their preceding contexts as 푃 (푤푁 1 ) = ∏푁 푖=1 푃(푤푖|푤푖−1 1 ). In 푛-gram LMs the unbounded conditional prob- abilities 푃 (푤푖|푤푖−1 1 ) are approximated by imposing a finite-order Markov assumption, 푃(푤푖|푤푖−1 1 ) ≈ 푃(푤푖|푤푖−1 푖−푛+1). Several smoothing techniques ad- dress the statistical sparsity issue for computing the conditional probabilities (Kneser and Ney, 1995; Chen and Goodman, 1999; Teh, 2006; Shareghi et al., 2016a), while others avoided the above ap- proximation with unbounded hierarchical nonpara- metric Bayesian frameworks (Wood et al., 2011; Shareghi et al., 2017). Alternatively, neural LMs compute 푃 (푤푖|푤푖−1 1 ) via recurrent neural units which, in theory, are ca- pable of encoding an unbounded context 푤푖−1 1 . In recent years, neural LMs have become the promi- nent class of language modeling and have estab- lished state-of-the-art results on almost all suffi- ciently large benchmarks (Melis et al., 2018; Yang et al., 2018). While outperforming 푛-grams in terms of predictive accuracy, the computational shortcom- ings of neural LMs are well-documented: Training neural LMs is computationally expensive to the point that running experiments on large data (≥ a few GiBs) is beyond the reach of academic …

Content truncated to 500 words