Softmax-Margin CRFs: Training Log-Linear Models with Cost Functions

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 733–736, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Softmax-Margin CRFs: Training Log-Linear Models with Cost Functions Kevin Gimpel Noah A. Smith School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213, USA {kgimpel,nasmith}@cs.cmu.edu Abstract We describe a method of incorporating task- specific cost functions into standard condi- tional log-likelihood (CLL) training of linear structured prediction models. Recently intro- duced in the speech recognition community, we describe the method generally for struc- tured models, highlight connections to CLL and max-margin learning for structured pre- diction (Taskar et al., 2003), and show that the method optimizes a bound on risk. The approach is simple, efficient, and easy to im- plement, requiring very little change to an existing CLL implementation. We present experimental results comparing with several commonly-used methods for training struc- tured predictors for named-entity recognition. 1 Introduction Conditional random fields (CRFs; Lafferty et al, 2001) and other conditional log-linear models (Berger et al., 1996) achieve strong performance for many NLP problems, but the conditional log- likelihood (CLL) criterion optimized when training these models cannot take a task-specific cost func- tion into account. In this paper, we describe a simple approach for training conditional log-linear models with cost functions. We show how the method relates to other methods and how it provides a bound on risk. We apply the method to train a discriminative model for named-entity recognition, showing a statistically significant improvement over CLL. 2 Structured Log-Linear Models Let X denote a structured input space and, for a par- ticular x ∈ X, let Y(x) denote a structured output space for x. The size of Y(x) is often exponential in x, which differentiates structured prediction from multiclass classification. For named-entity recogni- tion, for example, x might be a sentence and Y(x) the set of all possible named-entity labelings for the sentence. Given an x ∈ X and a y ∈ Y(x), we use a conditional log-linear model for pθ(y|x): pθ(y|x) = exp{θ⊤f(x, y)} � y′∈Y(x) exp{θ⊤f(x, y′)} (1) where f(x, y) is a feature vector representation of x and y and θ is a parameter vector containing one component for each feature. 2.1 Training Criteria Many criteria exist for training the weights θ. We next review three choices in detail. For the follow- ing, we assume a training set consisting of n exam- ples {⟨x(i), y(i)⟩}n i=1. Some criteria will make use of a task-specific cost function that measures the extent to which a structure y differs from the true structure y(i), denoted by cost(y(i), y). 2.1.1 Conditional Log-Likelihood The learning problem for maximizing conditional log-likelihood is shown in Eq. 3 in Fig. 1 (we trans- form it into a minimization problem for easier com- parison). This criterion is commonly used when a probabilistic interpretation of the model is desired. 2.1.2 Max-Margin An alternative approach to training structured lin- ear classifiers is based on maximum-margin Markov networks (Taskar et al., 2003). The …

Content truncated to 500 words