Richer Interpolative Smoothing Based on Modified Kneser-Ney Language Modeling

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 944–949, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Richer Interpolative Smoothing Based on Modified Kneser-Ney Language Modeling Ehsan Shareghi,♣ Trevor Cohn♠ and Gholamreza Haffari♣ ♣ Faculty of Information Technology, Monash University ♠ Computing and Information Systems, The University of Melbourne first.last@{monash.edu,unimelb.edu.au} Abstract In this work we present a generalisation of the Modified Kneser-Ney interpolative smoothing for richer smoothing via additional discount parameters. We provide mathematical under- pinning for the estimator of the new discount parameters, and showcase the utility of our rich MKN language models on several Euro- pean languages. We further explore the in- terdependency among the training data size, language model order, and number of dis- count parameters. Our empirical results illus- trate that larger number of discount parame- ters, i) allows for better allocation of mass in the smoothing process, particularly on small data regime where statistical sparsity is se- vere, and ii) leads to significant reduction in perplexity, particularly for out-of-domain test sets which introduce higher ratio of out-of- vocabulary words.1 1 Introduction Probabilistic language models (LMs) are the core of many natural language processing tasks, such as machine translation and automatic speech recogni- tion. m-gram models, the corner stone of language modeling, decompose the probability of an utter- ance into conditional probabilities of words given a fixed-length context. Due to sparsity of the events in natural language, smoothing techniques are criti- cal for generalisation beyond the training text when estimating the parameters of m-gram LMs. This is particularly important when the training text is 1For the implementation see: https://github.com/ eehsan/cstlm small, e.g. building language models for translation or speech recognition in low-resource languages. A widely used and successful smoothing method is interpolated Modified Kneser-Ney (MKN) (Chen and Goodman, 1999). This method uses a linear in- terpolation of higher and lower order m-gram prob- abilities by preserving probability mass via absolute discounting. In this paper, we extend MKN by in- troducing additional discount parameters, leading to a richer smoothing scheme. This is particularly im- portant when statistical sparsity is more severe, i.e., in building high-order LMs on small data, or when out-of-domain test sets are used. Previous research in MKN language modeling, and more generally m-gram models, has mainly dedicated efforts to make them faster and more com- pact (Stolcke et al., 2011; Heafield, 2011; Shareghi et al., 2015) using advanced data structures such as succinct suffix trees. An exception is Hierarchical Pitman-Yor Process LMs (Teh, 2006a; Teh, 2006b) providing a rich Bayesian smoothing scheme, for which Kneser-Ney smoothing corresponds to an ap- proximate inference method. Inspired by this work, we directly enrich MKN smoothing realising some of the reductions while remaining more efficient in learning and inference. We provide estimators for our additional discount parameters by extending the discount bounds in MKN. We empirically analyze our enriched MKN LMs on several European languages in in- and out- of-domain settings. The results show that our dis- counting mechanism significantly improves …

Content truncated to 500 words