Latent Tree Language Model

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 436–446, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Latent Tree Language Model Tom´aˇs Brychc´ın NTIS – New Technologies for the Information Society, Faculty of Applied Sciences, University of West Bohemia, Technick´a 8, 306 14 Plzeˇn, Czech Republic brychcin@kiv.zcu.cz nlp.kiv.zcu.cz Abstract In this paper we introduce Latent Tree Lan- guage Model (LTLM), a novel approach to language modeling that encodes syntax and semantics of a given sentence as a tree of word roles. The learning phase iteratively updates the trees by moving nodes according to Gibbs sampling. We introduce two algorithms to in- fer a tree for a given sentence. The first one is based on Gibbs sampling. It is fast, but does not guarantee to find the most probable tree. The second one is based on dynamic program- ming. It is slower, but guarantees to find the most probable tree. We provide comparison of both algorithms. We combine LTLM with 4-gram Modified Kneser-Ney language model via linear inter- polation. Our experiments with English and Czech corpora show significant perplexity re- ductions (up to 46% for English and 49% for Czech) compared with standalone 4-gram Modified Kneser-Ney language model. 1 Introduction Language modeling is one of the core disciplines in natural language processing (NLP). Automatic speech recognition, machine translation, optical character recognition, and other tasks strongly de- pend on the language model (LM). An improve- ment in language modeling often leads to better performance of the whole task. The goal of lan- guage modeling is to determine the joint probabil- ity of a sentence. Currently, the dominant approach is n-gram language modeling, which decomposes the joint probability into the product of conditional probabilities by using the chain rule. In traditional n-gram LMs the words are represented as distinct symbols. This leads to an enormous number of word combinations. In the last years many researchers have tried to capture words contextual meaning and incorporate it into the LMs. Word sequences that have never been seen before receive high probability when they are made of words that are semantically similar to words forming sentences seen in training data. This ability can increase the LM performance because it reduces the data sparsity problem. In NLP a very common paradigm for word meaning representation is the use of the Distributional hypothesis. It sug- gests that two words are expected to be semanti- cally similar if they occur in similar contexts (they are similarly distributed in the text) (Harris, 1954). Models based on this assumption are denoted as dis- tributional semantic models (DSMs). Recently, semantically motivated LMs have be- gun to surpass the ordinary n-gram LMs. The most commonly used architectures are neural network LMs (Bengio et al., 2003; Mikolov et al., 2010; Mikolov et al., 2011) and class-based LMs. Class- based LMs are more related to this work thus we investigate them deeper. Brown et al. (1992) introduced class-based LMs of English. Their unsupervised algorithm searches classes consisting of …

Content truncated to 500 words