Minimum-Risk Training of Approximate CRF-Based NLP Systems

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 120–130, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Minimum-Risk Training of Approximate CRF-Based NLP Systems Veselin Stoyanov and Jason Eisner HLTCOE and Center for Language and Speech Processing Johns Hopkins University Baltimore, MD 21218 {ves, jason}@cs.jhu.edu Abstract Conditional Random Fields (CRFs) are a pop- ular formalism for structured prediction in NLP. It is well known how to train CRFs with certain topologies that admit exact inference, such as linear-chain CRFs. Some NLP phe- nomena, however, suggest CRFs with more complex topologies. Should such models be used, considering that they make exact infer- ence intractable? Stoyanov et al. (2011) re- cently argued for training parameters to min- imize the task-specific loss of whatever ap- proximate inference and decoding methods will be used at test time. We apply their method to three NLP problems, showing that (i) using more complex CRFs leads to im- proved performance, and that (ii) minimum- risk training learns more accurate models. 1 Introduction Conditional Random Fields (CRFs) (Lafferty et al., 2001) are often used to model dependencies among linguistic variables. CRF-based models have im- proved the state of the art in a number of natural language processing (NLP) tasks ranging from part- of-speech tagging to information extraction and sen- timent analysis (Lafferty et al., 2001; Peng and Mc- Callum, 2006; Choi et al., 2005). Robust and theoretically sound training proce- dures have been developed for CRFs when the model can be used with exact inference and de- coding.1 However, some NLP problems seem to 1“Inference” typically refers to computing posterior marginal or max-marginal probability distributions of output random variables, given some evidence. “Decoding” derives a single structured output from the results of inference. call for higher-treewidth graphical models in which exact inference is expensive or intractable. These “loopy” CRFs have cyclic connections among the output and/or latent variables. Alas, standard learn- ing procedures assume exact inference: they do not compensate for approximations that will be used at test time, and can go surprisingly awry if approxi- mate inference is used at training time (Kulesza and Pereira, 2008). While NLP research has been consistently evolv- ing toward more richly structured models, one may hesitate to add dependencies to a graphical model if there is a danger that this will end up hurting per- formance through approximations. In this paper we illustrate how to address this problem, even for ex- tremely interconnected models in which every pair of output variables is connected. Wainwright (2006) showed that if approximate in- ference will be used at test time, it may be beneficial to use a learning procedure that does not converge to the true model but to one that performs well under the approximations. Stoyanov et al. (2011) argue for minimizing a certain non-convex training objective, namely the empirical risk of the entire system com- prising the CRF together with whatever approximate inference and decoding procedures will be used at test …

Content truncated to 500 words