Improvements to the Bayesian Topic N-Gram Models

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1180–1190, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Improvements to the Bayesian Topic N-gram Models Hiroshi Noji†‡ noji@nii.ac.jp Daichi Mochihashi†∗ daichi@ism.ac.jp †Graduate University for Advanced Studies ‡National Institute of Informatics, Tokyo, Japan ∗The Institute of Statistical Mathematics, Tokyo, Japan Yusuke Miyao†‡ yusuke@nii.ac.jp Abstract One of the language phenomena that n-gram language model fails to capture is the topic in- formation of a given situation. We advance the previous study of the Bayesian topic language model by Wallach (2006) in two directions: one, investigating new priors to alleviate the sparseness problem caused by dividing all n- grams into exclusive topics, and two, develop- ing a novel Gibbs sampler that enables moving multiple n-grams across different documents to another topic. Our blocked sampler can efficiently search for higher probability space even with higher order n-grams. In terms of modeling assumption, we found it is effective to assign a topic to only some parts of a docu- ment. 1 Introduction N-gram language model is still ubiquitous in NLP, but due to its simplicity it fails to capture some im- portant aspects of language, such as difference of word usage in different situations, sentence level syntactic correctness, and so on. Toward language model that can consider such a more global con- text, many extensions have been proposed from lexical pattern adaptation, e.g., adding cache (Je- linek et al., 1991) or topic information (Gildea and Hofmann, 1999; Wallach, 2006), to grammaticality aware models (Pauls and Klein, 2012). Topic language models are important for use in e.g., unsupervised language model adaptation: we want a language model that can adapt to the do- main or topic of the current situation (e.g., a doc- ument in SMT or a conversation in ASR) automat- ically and select the appropriate words using both topic and syntactic context. Wallach (2006) is one such model, which generate each word based on lo- cal context and global topic information to capture the difference of lexical usage among different top- ics. However, Wallach’s experiments were limited to bigrams, a toy setting for language models, and ex- periments with higher-order n-grams have not yet been sufficiently studied, which we investigate in this paper. In particular, we point out the two funda- mental problems caused when extending Wallach’s model to a higher-order: sparseness caused by di- viding all n-grams into exclusive topics, and local minima caused by the deep hierarchy of the model. On resolving these problems, we make several con- tributions to both computational linguistics and ma- chine learning. To address the first problem, we investigate incor- porating a global language model for ease of sparse- ness, along with some priors on a suffix tree to cap- ture the difference of topicality for each context, which include an unsupervised extension of the dou- bly hierarchical Pitman-Yor language model (Wood and Teh, 2009), a Bayesian generative model for su- pervised language model adaptation. For the sec- ond inference problem, …

Content truncated to 500 words