Trans-dimensional Random Fields for Language Modeling

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 785–794, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Trans-dimensional Random Fields for Language Modeling Bin Wang1, Zhijian Ou1, Zhiqiang Tan2 1Department of Electronic Engineering, Tsinghua University, Beijing 100084, China 2Department of Statistics, Rutgers University, Piscataway, NJ 08854, USA wangbin12@mails.tsinghua.edu.cn, ozj@tsinghua.edu.cn, ztan@stat.rutgers.edu Abstract Language modeling (LM) involves determining the joint probability of words in a sentence. The conditional approach is dominant, representing the joint probability in terms of conditionals. Examples include n-gram LMs and neural network LMs. An alternative approach, called the random field (RF) approach, is used in whole-sentence maximum entropy (WSME) LMs. Although the RF approach has potential benefits, the empirical results of previous WSME models are not satisfactory. In this paper, we revisit the RF approach for language modeling, with a number of innovations. We propose a trans-dimensional RF (TDRF) model and develop a training algorithm using joint stochastic approximation and trans-dimensional mixture sampling. We perform speech recognition experiments on Wall Street Journal data, and find that our TDRF models lead to performances as good as the recurrent neural network LMs but are computationally more efficient in computing sentence probability. 1 Introduction Language modeling is crucial for a variety of computational linguistic applications, such as speech recognition, machine translation, handwriting recognition, information retrieval and so on. It involves determining the joint probability p(x) of a sentence x, which can be denoted as a pair x = (l, xl), where l is the length and xl = (x1, . . . , xl) is a sequence of l words. Currently, the dominant approach is conditional modeling, which decomposes the joint probability of xl into a product of conditional probabilities 1 1And the joint probability of x is modeled as p(x) = by using the chain rule, p(x1, . . . , xl) = l� i=1 p(xi|x1, . . . , xi−1). (1) To avoid degenerate representation of the con- ditionals, the history of xi, denoted as hi (x1, · · · , xi−1), is reduced to equivalence classes= through a mapping φ(hi) with the assumption p(xi|hi) ≈ p(xi|φ(hi)). (2) Language modeling in this conditional approach consists of finding suitable mappings φ(hi) and effective methods to estimate p(xi|φ(hi)). A classic example is the traditional n-gram LMs with φ(hi) = (xi−n+1, . . . , xi−1). Various smoothing techniques are used for parameter estimation (Chen and Goodman, 1999). Recently, neural network LMs, which have begun to surpass the traditional n-gram LMs, also follow the conditional modeling approach, with φ(hi) determined by a neural network (NN), which can be either a feedforward NN (Schwenk, 2007) or a recurrent NN (Mikolov et al., 2011). Remarkably, an alternative approach is used in whole-sentence maximum entropy (WSME) lan- guage modeling (Rosenfeld et al., 2001). Specifi- cally, a WSME model has the form: p(x; λ) = 1 Z exp{λT f(x)} (3) Here f(x) is a vector of features, which can …

Content truncated to 500 words