Optimizing Multivariate Performance Measures for Learning Relation Extraction Models

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 892–900, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Optimizing Multivariate Performance Measures for Learning Relation Extraction Models Gholamreza Haffari 2Faculty of IT, Monash University gholamreza.haffari@monash.edu Ajay Nagesh1,2,3 1IITB-Monash Research Academy ajaynagesh@cse.iitb.ac.in Ganesh Ramakrishnan 3Dept. of CSE, IIT Bombay ganesh@cse.iitb.ac.in Abstract We describe a novel max-margin learn- ing approach to optimize non-linear perfor- mance measures for distantly-supervised re- lation extraction models. Our approach can be generally used to learn latent variable models under multivariate non-linear perfor- mance measures, such as Fβ-score. Our approach interleaves Concave-Convex Pro- cedure (CCCP) for populating latent vari- ables with dual decomposition to factorize the original hard problem into smaller inde- pendent sub-problems. The experimental re- sults demonstrate that our learning algorithm is more effective than the ones commonly used in the literature for distant supervision of in- formation extraction models. On several data conditions, we show that our method outper- forms the baseline and results in up to 8.5% improvement in the F1-score. 1 Introduction Rich models with latent variables are popular for many problems in natural language processing. In information extraction, for example, one needs to predict the relation labels y that an entity-pair x can have based on the hidden relation mentions h, i.e., the relation labels for occurrences of the entity-pair in a given corpus. However, these models are often trained by optimizing performance measures (such as conditional log-likelihood or error rate) that are not directly related to the task-specific non-linear performance measure, e.g., the F1-score. However, better models may be trained by optimizing the task- specific performance measure while allowing latent variables to adapt their values accordingly. We present a large-margin method to learn pa- rameters of latent variable models for a wide range of non-linear multivariate performance mea- sures such as Fβ. Our method can be applied to learning graphical models that incorporate inter- dependencies among the output variables either di- rectly, or indirectly through hidden variables. Large-margin methods have been shown to be a compelling approach to learn rich models detailing the inter-dependencies among the output variables, via optimizing loss functions decomposable over the training instances (Taskar et al., 2003; Tsochan- taridis et al., 2004) or non-decompasable loss func- tions (Ranjbar et al., 2013; Tarlow and Zemel, 2012; Rosenfeld et al., 2014; Keshet, 2014). They have also been shown to be powerful when applied to la- tent variable models when optimizing for decompos- able loss functions (Wang and Mori, 2011; Felzen- szwalb et al., 2010; Yu and Joachims, 2009). Our large-margin method learns latent variable models via optimizing non-decomposable loss func- tions. It interleaves the Concave-Convex Procedure (CCCP) (Yuille and Rangarajan, 2001) for populat- ing latent variables with dual decomposition (Ko- modakis et al., 2011; Rush and Collins, 2012). The latter factorizes the hard optimization problem (encountered in learning) into smaller independent sub-problems over the training instances. We then present linear programming and local search meth- ods for …

Content truncated to 500 words