Joint and Conditional Estimation of Tagging and Parsing Models

Joint and conditional estimation of tagging and parsing models∗ Mark Johnson Brown University Mark Johnson@Brown.edu Abstract This paper compares two different ways of estimating statistical language mod- els. Many statistical NLP tagging and parsing models are estimated by max- imizing the (joint) likelihood of the fully-observed training data. How- ever, since these applications only re- quire the conditional probability distri- butions, these distributions can in prin- ciple be learnt by maximizing the con- ditional likelihood of the training data. Perhaps somewhat surprisingly, models estimated by maximizing the joint were superior to models estimated by max- imizing the conditional, even though some of the latter models intuitively had access to “more information”. 1 Introduction Many statistical NLP applications, such as tag- ging and parsing, involve finding the value of some hidden variable Y (e.g., a tag or a parse tree) which maximizes a conditional prob- ability distribution Pθ(Y |X), where X is a given word string. The model parameters θ are typically estimated by maximum likelihood: i.e., maximizing the likelihood of the training ∗I would like to thank Eugene Charniak and the other members of BLLIP for their comments and suggestions. Fer- nando Pereira was especially generous with comments and suggestions, as were the ACL reviewers; I apologize for not being able to follow up all of your good suggestions. This re- search was supported by NSF awards 9720368 and 9721276 and NIH award R01 MH60922-01A2. data. Given a (fully observed) training cor- pus D = ((y1, x1), . . . , (yn, xn)), the maximum (joint) likelihood estimate (MLE) of θ is: ˆθ = argmax θ n � i=1 Pθ(yi, xi). (1) However, it turns out there is another maximum likelihood estimation method which maximizes the conditional likelihood or “pseudo-likelihood” of the training data (Besag, 1975). Maximum conditional likelihood is consistent for the con- ditional distribution. Given a training corpus D, the maximum conditional likelihood estimate (MCLE) of the model parameters θ is: ˆθ = argmax θ n � i=1 Pθ(yi|xi). (2) Figure 1 graphically depicts the difference be- tween the MLE and MCLE. Let Ω be the universe of all possible pairs (y, x) of hidden and visible values. Informally, the MLE selects the model parameter θ which make the training data pairs (yi, xi) as likely as possible relative to all other pairs (y′, x′) in Ω. The MCLE, on the other hand, selects the model parameter θ in order to make the training data pair (yi, xi) more likely than other pairs (y′, xi) in Ω, i.e., pairs with the same visible value xi as the training datum. In statistical computational linguistics, max- imum conditional likelihood estimators have mostly been used with general exponential or “maximum entropy” models because standard maximum likelihood estimation is usually com- putationally intractable (Berger et al., 1996; Della Pietra et al., 1997; Jelinek, 1997). Well- known computational linguistic models such as (MLE) (MCLE) Ω Y = yi, X = xi Ω X = xi Y = yi, X = xi Figure 1: …

Content truncated to 500 words