Logarithmic Opinion Pools for Conditional Random Fields

Proceedings of the 43rd Annual Meeting of the ACL, pages 18–25, Ann Arbor, June 2005. c⃝2005 Association for Computational Linguistics Logarithmic Opinion Pools for Conditional Random Fields Andrew Smith Division of Informatics University of Edinburgh United Kingdom a.p.smith-2@sms.ed.ac.uk Trevor Cohn Department of Computer Science and Software Engineering University of Melbourne, Australia tacohn@csse.unimelb.edu.au Miles Osborne Division of Informatics University of Edinburgh United Kingdom miles@inf.ed.ac.uk Abstract Recent work on Conditional Random Fields (CRFs) has demonstrated the need for regularisation to counter the tendency of these models to overfit. The standard approach to regularising CRFs involves a prior distribution over the model parame- ters, typically requiring search over a hy- perparameter space. In this paper we ad- dress the overfitting problem from a dif- ferent perspective, by factoring the CRF distribution into a weighted product of in- dividual “expert” CRF distributions. We call this model a logarithmic opinion pool (LOP) of CRFs (LOP-CRFs). We ap- ply the LOP-CRF to two sequencing tasks. Our results show that unregularised expert CRFs with an unregularised CRF under a LOP can outperform the unregularised CRF, and attain a performance level close to the regularised CRF. LOP-CRFs there- fore provide a viable alternative to CRF regularisation without the need for hyper- parameter search. 1 Introduction In recent years, conditional random fields (CRFs) (Lafferty et al., 2001) have shown success on a num- ber of natural language processing (NLP) tasks, in- cluding shallow parsing (Sha and Pereira, 2003), named entity recognition (McCallum and Li, 2003) and information extraction from research papers (Peng and McCallum, 2004). In general, this work has demonstrated the susceptibility of CRFs to over- fit the training data during parameter estimation. As a consequence, it is now standard to use some form of overfitting reduction in CRF training. Recently, there have been a number of sophisti- cated approaches to reducing overfitting in CRFs, including automatic feature induction (McCallum, 2003) and a full Bayesian approach to training and inference (Qi et al., 2005). These advanced meth- ods tend to be difficult to implement and are of- ten computationally expensive. Consequently, due to its ease of implementation, the current standard approach to reducing overfitting in CRFs is the use of a prior distribution over the model parameters, typically a Gaussian. The disadvantage with this method, however, is that it requires adjusting the value of one or more of the distribution’s hyper- parameters. This usually involves manual or auto- matic tuning on a development set, and can be an ex- pensive process as the CRF must be retrained many times for different hyperparameter values. In this paper we address the overfitting problem in CRFs from a different perspective. We factor the CRF distribution into a weighted product of indi- vidual expert CRF distributions, each focusing on a particular subset of the distribution. We call this model a logarithmic opinion pool (LOP) of CRFs (LOP-CRFs), and provide a procedure for learning the weight of each expert in the product. The LOP- CRF framework is “parameter-free” in the sense that it …

Content truncated to 500 words