Training Conditional Random Fields with Multivariate Evaluation Measures

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 217–224, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Training Conditional Random Fields with Multivariate Evaluation Measures Jun Suzuki, Erik McDermott and Hideki Isozaki NTT Communication Science Laboratories, NTT Corp. 2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0237 Japan {jun, mcd, isozaki}@cslab.kecl.ntt.co.jp Abstract This paper proposes a framework for train- ing Conditional Random Fields (CRFs) to optimize multivariate evaluation mea- sures, including non-linear measures such as F-score. Our proposed framework is derived from an error minimization ap- proach that provides a simple solution for directly optimizing any evaluation mea- sure. Specifically focusing on sequential segmentation tasks, i.e. text chunking and named entity recognition, we introduce a loss function that closely reflects the tar- get evaluation measure for these tasks, namely, segmentation F-score. Our ex- periments show that our method performs better than standard CRF training. 1 Introduction Conditional random fields (CRFs) are a recently introduced formalism (Lafferty et al., 2001) for representing a conditional model p(y|x), where both a set of inputs, x, and a set of outputs, y, display non-trivial interdependency. CRFs are basically defined as a discriminative model of Markov random fields conditioned on inputs (ob- servations) x. Unlike generative models, CRFs model only the output y’s distribution over x. This allows CRFs to use flexible features such as com- plicated functions of multiple observations. The modeling power of CRFs has been of great ben- efit in several applications, such as shallow pars- ing (Sha and Pereira, 2003) and information ex- traction (McCallum and Li, 2003). Since the introduction of CRFs, intensive re- search has been undertaken to boost their effec- tiveness. The first approach to estimating CRF pa- rameters is the maximum likelihood (ML) criterion over conditional probability p(y|x) itself (Laf- ferty et al., 2001). The ML criterion, however, is prone to over-fitting the training data, espe- cially since CRFs are often trained with a very large number of correlated features. The maximum a posteriori (MAP) criterion over parameters, λ, given x and y is the natural choice for reducing over-fitting (Sha and Pereira, 2003). Moreover, the Bayes approach, which optimizes both MAP and the prior distribution of the parameters, has also been proposed (Qi et al., 2005). Furthermore, large margin criteria have been employed to op- timize the model parameters (Taskar et al., 2004; Tsochantaridis et al., 2005). These training criteria have yielded excellent re- sults for various tasks. However, real world tasks are evaluated by task-specific evaluation mea- sures, including non-linear measures such as F- score, while all of the above criteria achieve op- timization based on the linear combination of av- erage accuracies, or error rates, rather than a given task-specific evaluation measure. For example, se- quential segmentation tasks (SSTs), such as text chunking and named entity recognition, are gener- ally evaluated with the segmentation F-score. This inconsistency between the objective function dur- ing training and the task evaluation measure might produce a suboptimal result. In fact, to overcome …

Content truncated to 500 words