Conditional Structure versus Conditional Estimation in NLP Models

Conditional Structure versus Conditional Estimation in NLP Models Dan Klein and Christopher D. Manning Computer Science Department Stanford University Stanford, CA 94305-9040 {klein, manning}@cs.stanford.edu Abstract This paper separates conditional parameter estima- tion, which consistently raises test set accuracy on statistical NLP tasks, from conditional model struc- tures, such as the conditional Markov model used for maximum-entropy tagging, which tend to lower accuracy. Error analysis on part-of-speech tagging shows that the actual tagging errors made by the conditionally structured model derive not only from label bias, but also from other ways in which the in- dependence assumptions of the conditional model structure are unsuited to linguistic sequences. The paper presents new word-sense disambiguation and POS tagging experiments, and integrates apparently conflicting reports from other recent work. 1 Introduction The success and widespread adoption of probabilis- tic models in NLP has led to numerous variant meth- ods for any given task, and it can be difficult to tell what aspects of a system have led to its relative suc- cesses or failures. As an example, maximum en- tropy taggers have achieved very good performance (Ratnaparkhi, 1998; Toutanova and Manning, 2000; Lafferty et al., 2001), but almost identical perfor- mance has also come from finely tuned HMM mod- els (Brants, 2000; Thede and Harper, 1999). Are any performance gains due to the sequence model used, the maximum entropy approach to parameter estima- tion, or the features employed by the system? Recent experiments have given conflicting recom- mendations. Johnson (2001) finds that a condition- ally trained PCFG marginally outperforms a standard jointly trained PCFG, but that a conditional shift- reduce model performs worse than a joint formu- lation. Lafferty et al. (2001) suggest on abstract grounds that conditional models will suffer from a phenomenon called label bias (Bottou, 1991) – see section 3 – but is this a significant effect for real NLP problems? We suggest that the results in the literature, along with the new results we present in this work, can be explained by the following generalizations: • The ability to include better features in a well- founded fashion leads to better performance. • For fixed features, assumptions implicit in the model structure have a large impact on errors. • Maximizing the objective being evaluated has a re- liably positive, but often small, effect. It is especially important to study these issues us- ing NLP data sets: NLP tasks are marked by their complexity and sparsity, and, as we show, conclu- sions imported from the machine-learning literature do not always hold in these characteristic contexts. In previous work, the structure of a model and the method of parameter estimation were often both changed simultaneously (for reasons of naturalness or computational ease), but in this paper we seek to tease apart the separate effects of these two factors. In section 2, we take the Naive-Bayes model, ap- plied to word-sense disambiguation (WSD), and train it to maximize various objective functions. Our ex- periments reaffirm that discriminative objectives like conditional likelihood improve test-set accuracy. …

Content truncated to 500 words