Bayesian Language Model based on Mixture of Segmental Contexts for Spontaneous Utterances with Unexpected Words

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 161–170, Osaka, Japan, December 11-17 2016. Bayesian Language Model based on Mixture of Segmental Contexts for Spontaneous Utterances with Unexpected Words Ryu Takeda Kazunori Komatani The Institute of Scientific and Industrial Research, Osaka University 8-1, Mihogaoka, Ibaraki, Osaka 567-0047, Japan {rtakeda, komatani}@sanken.osaka-u.ac.jp Abstract This paper describes a Bayesian language model for predicting spontaneous utterances. People sometimes say unexpected words, such as fillers or hesitations, that cause the miss-prediction of words in normal N-gram models. Our proposed model considers mixtures of possible segmental contexts, that is, a kind of context-word selection. It can reduce negative effects caused by un- expected words because it represents conditional occurrence probabilities of a word as weighted mixtures of possible segmental contexts. The tuning of mixture weights is the key issue in this ap- proach as the segment patterns becomes numerous, thus we resolve it by using Bayesian model. The generative process is achieved by combining the stick-breaking process and the process used in the variable order Pitman-Yor language model. Experimental evaluations revealed that our model outperformed contiguous N-gram models in terms of perplexity for noisy text including hesitations. 1 Introduction 1.1 Background Language models (LMs) are widely used for text analysis, word segmentation and word prediction in automatic speech recognition (ASR). The basic LM is a conventional N-gram model that predicts a word depending on the patterns of the previous N words (context). The probability of a word is usually calculated by counting the words that match the context in text data as maximum likelihood estimation. Therefore, the model easily predicts frequent words or set expressions but not rare words or phrases. Various N-gram language models have been proposed to prevent the incorrect probability assignment caused by the increase of the context length N. Since the number of combinations of N becomes O(V N) for vocabulary size V , there are a lot of patterns that do not appear in training data (data sparseness). Us- ing an N-gram model based on a Bayesian framework is a promising approach for data sparseness. Be- cause it is based on a Bayesian framework, an LM based on hierarchical Pitman-Yor process (HPYLM) has two main differences from previous language models (Teh, 2006), such as Witten-bell (WB) (Witten and Bell, 1991) and Kneser-ney (KN) smoothing (Kneser and Ney, 1995): 1) a Bayesian model express- ing conventional smoothing methods and 2) automatic tuning of parameters from data. Since HPYLM is based on a Bayesian framework, we can integrate other probabilistic models theoretically for other problems and apply optimization methods in accordance with a Bayesian framework. In contrast, other smoothing methods has several parameters that need to be tuned manually. Human utterances contain various fillers and hesitations (left in Fig. 1), and these cause the mis- prediction of words because they rarely appear in the training data, that is, another type of sparsity. This will affect 1) the word prediction accuracy in ASR and 2) the precision of …

Content truncated to 500 words