MAP Estimation of Continuous Density HMM : Theory and Applications

MAP Estimation of Continuous Density HMM : Theory and Applications Jean-Luc Gauvain t and Chin-Hui Lee Speech Research Department AT&T Bell Laboratories Murray Hill, NJ 07974 ABSTRACT We discuss maximum a posteriori estimation of continuous density hid- den Markov models (CDHMM). The classical MLE reestimation algorithms, namely the forward-backward algorithm and the segmental k-means algo- rithm, are expanded and reestimation formulas are given for HMM with Gaussian mixture observation densities. Because of its adaptive nature, Bayesian learning serves as a unified approach for the following four speech recognition applications, namely parameter smoothing, speaker adaptation, speaker group modeling and corrective ~aining. New experimental results on all four applications are provided to show the effectiveness of the MAP estimation approach. INTRODUCTION Estimation of hidden Marknv model (HMM) is usually obtained by the method of maximum likelihood (ML) [1, 10, 6] assuming that the size of the training data is large enough to provide robust estimates. This paper investigates maximum a posteriori (MAP) estimate of continuous density hidden Markov models (CDHMM). The MAP estimate can be seen as a Bayes estimate of the vector pa- rameter when the loss function is not specified [2]. This estimation technique provides a way of incorporatimg prior information in the training process, which is particularly useful to deal with problems posed by sparse training data for which the ML approach gives inaccurate estimates. This approach can be applied to two classes of estimation problems, namely, parameter smoothing and model adaptation, both related to the problem of sparse training data. In the following the sample x = (zl, ...,z,~) is a given set of n observations, where zl, ..., z n are either independent and identi- cally distributed (i.i.d.), or are drawn from a probabilistic function of a Markov chain. The difference between MAP and ML estimation lies in the assumption of an appropriate prior disliibution of the parameters to be estimated. If 0, assumed to be a random vector taking values in the space O, is the parameter vector to be estimated from the sample x with probability density function (p.d.f.) f(.lO), and if g is the prior p.d.f, of 0, then the MAP estimate, 0~p, is defined as the mode of the posterior p.d.f, of 0, i.e. Oma, = argmoax f(xlO)g(O) (I) If 9 is assumed to be fixed but unknown, then there is no knowl- edge about 8, which is equivalent to assuming a non-informative improper prior, i,e. g(8) ----constant. Equation (1) then reduces to the familiar ML formulation. Given the MAP formulation two problems remain: the choice of the prior distribution family and the evaluation of the maximum a ~This work was done while Jean-Luc Gauvain was on leave from the Speech Communication Group at LIMSI/CNRS, Orsay, France. posteriori. These two problems are closely related, since the appro- pilate choice of the prior distribution can greatly simplify the MAP estimation. Like for ML estimation, MAP estimation is relatively easy if the famay ofp.d.f.'s {f(-10), 0 ~ O} possesses a sufficient statistic of fixed dimension …

Content truncated to 500 words