Regularisation Techniques for Conditional Random Fields: Parameterised Versus Parameter-Free

Regularisation Techniques for Conditional Random Fields: Parameterised Versus Parameter-Free Andrew Smith and Miles Osborne School of Informatics, University of Edinburgh, United Kingdom a.p.smith-2@sms.ed.ac.uk, miles@inf.ed.ac.uk Abstract. Recent work on Conditional Random Fields (CRFs) has demonstrated the need for regularisation when applying these models to real-world NLP data sets. Conventional approaches to regularising CRFs has focused on using a Gaussian prior over the model parameters. In this paper we explore other possibilities for CRF regularisation. We examine alternative choices of prior distribution and we relax the usual simplifying assumptions made with the use of a prior, such as constant hyperparameter values across features. In addition, we contrast the effec- tiveness of priors with an alternative, parameter-free approach. Specifi- cally, we employ logarithmic opinion pools (LOPs). Our results show that a LOP of CRFs can outperform a standard unregularised CRF and attain a performance level close to that of a regularised CRF, without the need for intensive hyperparameter search. 1 Introduction Recent work on Conditional Random Fields (CRFs) has demonstrated the need for regularisation when applying these models to real-world NLP data sets ([8], [9]). Standard approaches to regularising CRFs, and log-linear models in general, has focused on the use of a Gaussian prior. Typically, for simplicity, this prior is assumed to have zero mean and constant variance across model parameters. To date, there has been little work exploring other possibilities. One exception is Peng & McCallum [8]. They investigated feature-dependent variance for a Gaus- sian prior, and explored different families of feature sets. They also compared different priors for CRFs on an information extraction task. In the first part of this paper, we compare priors for CRFs on standard sequence labelling tasks in NLP: NER and POS tagging. Peng & McCallum used variable hyperparameter values only for a Gaussian prior, based on feature counts in the training data. We use an alternative Bayesian approach to mea- sure confidence in empirical expected feature counts, and apply this to all the priors we test. We also look at varying the Gaussian prior mean. Our results show that: (1) considerable search is required to identify good hyperparameter values for all priors (2) for optimal hyperparameter values, the priors we tested perform roughly equally well (3) in some cases performance can be improved using feature-dependent hyperparameter values. R. Dale et al. (Eds.): IJCNLP 2005, LNAI 3651, pp. 896–907, 2005. c ⃝ Springer-Verlag Berlin Heidelberg 2005 Regularisation Techniques for Conditional Random Fields 897 As can be seen, a significant short-coming of using priors for CRF regular- isation is the requirement for intensive search of hyperparameter space. In the second part of the paper we contrast this parameterised prior approach with an alternative, parameter-free method. We factor the CRF distribution into a weighted product of individual expert CRF distributions, each focusing on a particular subset of the distribution. We call this model a logarithmic opinion pool (LOP) of CRFs (LOP-CRFs). Our results show that LOP-CRFs, which are unregularised, can outperform the unregularised standard CRF and attain a performance …

Content truncated to 500 words