Competitive generative models with structure learning for NLP classification tasks

Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pages 576–584, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Competitive generative models with structure learning for NLP classification tasks Kristina Toutanova Microsoft Research Redmond, WA kristout@microsoft.com Abstract In this paper we show that generative models are competitive with and some- times superior to discriminative models, when both kinds of models are allowed to learn structures that are optimal for dis- crimination. In particular, we compare Bayesian Networks and Conditional log- linear models on two NLP tasks. We ob- serve that when the structure of the gen- erative model encodes very strong inde- pendence assumptions (a la Naive Bayes), a discriminative model is superior, but when the generative model is allowed to weaken these independence assumptions via learning a more complex structure, it can achieve very similar or better perfor- mance than a corresponding discrimina- tive model. In addition, as structure learn- ing for generative models is far more ef- ficient, they may be preferable for some tasks. 1 Introduction Discriminative models have become the models of choice for NLP tasks, because of their ability to easily incorporate non-independent features and to more directly optimize classification accuracy. State of the art models for many NLP tasks are ei- ther fully discriminative or trained using discrim- inative reranking (Collins, 2000). These include models for part-of-speech tagging (Toutanova et al., 2003), semantic-role labeling (Punyakanok et al., 2005; Pradhan et al., 2005b) and Penn Tree- bank parsing (Charniak and Johnson, 2005). The superiority of discriminative models has been shown on many tasks when the discrimina- tive and generative models use exactly the same model structure (Klein and Manning, 2002). How- ever, the advantage of the discriminative mod- els can be very slight (Johnson, 2001) and for small training set sizes generative models can be better because they need fewer training sam- ples to converge to the optimal parameter setting (Ng and Jordan, 2002). Additionally, many dis- criminative models use a generative model as a base model and add discriminative features with reranking (Collins, 2000; Charniak and Johnson, 2005; Roark et al., 2004), or train discriminatively a small set of weights for features which are gener- atively estimated probabilities (Raina et al., 2004; Och and Ney, 2002). Therefore it is important to study generative models and to find ways of mak- ing them better even when they are used only as components of discriminative models. Generative models may often perform poorly due to making strong independence assumptions about the joint distribution of features and classes. To avoid this problem, generative models for NLP tasks have often been manually designed to achieve an appropriate representation of the joint distribution, such as in the parsing models of (Collins, 1997; Charniak, 2000). This shows that when the generative models have a good model structure, they can perform quite well. In this paper, we look differently at compar- ing generative and discriminative models. We ask the question: given the same set of input features, …

Content truncated to 500 words